Open to Open Source

Financial firms have grown more comfortable with the use of open source in recent years. Dan DeFrancesco speaks to those on the buy and sell side about the necessary steps they take to put their firm in the best position to succeed using open source, and what the future holds for the space.

About four times a year, Northern Trust’s top technology executives travel to Palo Alto, Calif., to meet with venture capitalists. The Chicago-based asset manager’s CTO Scott Murray; director of architecture and innovation Len Hardy; head of infrastructure engineering and operations Vijay Luthra; and chief information security officer (CISO) Steve Locke all journey west to one of the biggest hubs for innovation. 

The trip, according to Hardy, gives the firm an opportunity to see some of the latest up-and-coming pieces of software. When you’re one of the largest buy-side firms in the world, with $875 billion in assets under management, you need to keep your finger on the pulse of the latest potential innovation opportunities. 

By heading to Silicon Valley, Northern Trust can see which new technology projects relate to its business, and decide if it wants to bring any into the firm for proofs-of-concept, and eventually incorporate them into its architecture. 

Throughout the process, the tech heads meet with venture capitalists (VCs) who provide an enterprise view on the entire fintech ecosystem, Hardy says. The VCs, due to their prominent place in the region, understand if something is useful and valid in the space.

“We see the capability gap narrowing; the open-source community is creating more capabilities that we can access today. Also, it’s tough to compete with how inexpensive open source is, and that price differential is going to start to play a part.” David Reilly, Bank of America

It was during one of those trips that a venture capitalist spoke to Hardy about where all the new technology was coming from. 

“One of the VCs basically said point blank that all of the new innovation in Silicon Valley is happening in the open-source space. There is not a lot that is going on in the pure commercially vended perspective,” Hardy recalls.

Whether it is software based in open source, such as Apache Hadoop or Docker, or something developed behind closed doors before being brought into the open-source space for additional innovation, open-source communities are where some of the greatest strides are being made in technology.

“I think all or most of the innovation we’re seeing, especially a lot of the big stuff, is happening in this sort of open source or open-source core model,” Hardy says. “I think in a lot of people’s minds, mine included, open source and innovation go hand-in-hand.”

Opposites Attract

At first glance, it seems open source would not be a good fit for financial services. The name alone appears to be unsuited for the capital markets. Banks, asset managers and hedge funds have always valued secrecy and gaining any advantage they can over competitors. 

Open source, on the other hand, thrives on contributions. The strongest projects are considered the ones with the most active communities, all with several contributors looking to update and improve the base code. It is a true group effort, with everyone working toward creating the best software possible. The philosophy is as the saying goes: “A rising tide lifts all boats.” 

“Open source encourages an environment in which people post their code,” says Elliot Noma, founder and managing director of New York-based hedge fund Garrett Asset Management. “It’s more than just the code itself. It creates an environment in which there are a lot of tools and analysis, and lots of discussion about what the issues with the tools are and how to improve them.”

And yet, here we are. Two sides with seemingly opposing viewpoints coming together. On both the buy side and sell side it seems everyone is at least dipping their toe in the open-source waters. 

David Reilly, Bank of America’s CTO, says one of the bank’s two stacks for its software-defined infrastructure (SDI) is open source. The decision shows the firm’s long-held belief in open source, as Reilly says that by 2018, BofA hopes to have 80 percent of its technology workloads on SDI. And while a majority of workloads on SDI currently sit on the other stack, which is built on a proprietary third-party platform, the goal is to move more workloads to the open-source stack, Reilly says.

“We see the capability gap narrowing; the open-source community is creating more capabilities that we can access today,” Reilly says. “Also, it’s tough to compete with how inexpensive open source is, and that price differential is going to start to play a part.”

The open-source movement doesn’t appear to have an end in sight. John Weir, head of BNY Mellon’s application platform and services (APS) group, says firms’ use of open source is here to stay. “I think open source is fundamentally part of the ecosystem for software development,” Weir says. “I cannot imagine a world where the only software you could use was that provided by vendors. I can never imagine going back to that.” 

Know Your Community

There are plenty of keys to effectively using open source, but the biggest that continues to crop up is the importance of picking the right communities. How to choose which piece of open-source software to use, and who are the core contributors of that project, could be the difference between integrating an effective new application and wasting three months on a dead-end project.

LedgerX, an institutional trading and clearing platform for bitcoin, is quite familiar with the process. Zach Dexter, CTO of LedgerX, says the exchange was built from the ground up using open source. No proprietary software packages from vendors were used during the exchange’s development; it was all in-house code and open-source software, according to Dexter.

Dexter says it’s important during the selection process to look at the core contributors of the project and analyze their history in the open-source community. Have they abandoned software projects before? If so, have they implemented transition plans? Have they had a reasonable depreciation schedule for the old software they’ve worked on? 

A firm should also look at the latest research in the field the firm is looking at, Dexter adds. If a project doesn’t fall in line with an industry trend, the project will likely lose funding, meaning an eventual loss of contributors. 

It’s an exhaustive project without a clear answer, but one that is necessary, especially for firms without a depth of understanding in the field.

“There is a risk that if you don’t have the right amount of skill and experience you’re going to pick the project that’s going to go out of date, or the direction it takes will be out of fashion in two to four months and then you’re going to have to rewrite your entire infrastructure or be stuck with a half-baked piece of software,” Dexter says.

Governance

Heavy usage of open source might be appropriate for a start-up that has flexibility thanks to its lack of legacy technology and need to make use of a smaller budget, but what about larger, established firms that have a large technology stack and more to lose over a project gone awry? 

As a major player in the asset management space, Northern Trust certainly has a lot at stake as it wades into the open-source waters. That’s why the firm has specific governance models in place depending on the type of application or platform it is looking to implement through open source, according to Hardy.

For pieces of software that are core to the enterprise, such as databases, application servers or container technology, Northern Trust will only use open source that is backed by vendor support, Hardy says. Implementing software that will be used across the business means much is at risk, and the firm wants to have vendor support for things like vulnerability patching or release management.

However, when it comes to open source for what Hardy calls, “application development flexibility and efficiency,” the firm has an entirely different governance model. For these tools, such as user-interface tooling, testing tools, application frameworks or development tools, the firm follows similar protocols outlined by LedgerX’s Dexter, inspecting the community and core contributors. 

All open-source software that falls in the second category is also run through Northern Trust’s Continuous Integration environment, according to Hardy. This allows the firm to see if the code is overly complex and has unused sections or security vulnerabilities. Developers are notified via a dashboard where they need to start looking and how they should approach it, Hardy says. 

“If we were going to bring in an open-source product that is in app development, the minute we integrate and build that first application using that code we’re going to get a really good picture about the quality of that code, potential security issues that code might have, and the complexity of the code because of the static code-scanning capabilities of our Continuous Integration environment,” Hardy says. “We use that as a tool to determine if we want to continue using this specific open-source product.”

Commitment Is Key

The success of a firm implementing an open-source project isn’t just dependent on selecting a proper community. A firm also needs to stay committed to the software, BNY Mellon’s Weir points out. 

The New York-based bank can attest to that. Weir says over the last few years his firm has migrated from being a very vendor-oriented organization to one that is much more in favor of an approach that embraces open source across almost everything it does. Weir, whose focus is mainly in the middleware parts of the stack, says his space accounts for the largest amount of open source. 

Applications built for use in the financial world can often last 15 to 20 years, Weir says. That should be kept in mind when looking at open source. Firms that have negative experiences with open source don’t treat it the same way they would a vendor-backed product, which is a mistake, according to Weir.

“You need to be able to actually build it, run the integration tests and, if necessary, take over ownership for it within your company as if you had written it yourself,” he says. 

Firms must consider that these open-source projects can become integral parts of the infrastructure and need to understand how to migrate away from the project when it’s time to sunset a system.

“You have to treat it just like you’d have treated any other piece of software you have acquired and to manage that and make it happen,” he says. “Organizations that do not plan for that and view open source as if it was free software—and therefore treat it as if it was disposable—are the ones that end up suffering.”

Contributing Conundrum 

What the future holds for open source in financial services lies not so much in the pure technology as the licenses backing them up. While several of those interviewed for this story had a hard time pinpointing any real disadvantages to using open source, licenses are a real issue, especially for those at larger firms. 

Weir spent almost two decades at Goldman Sachs before joining BNY Mellon in September 2015, and the New York-based investment bank has a well-publicized history with open source. 

First there was the complicated case surrounding former Goldman programmer Sergey Aleynikov. The bank claimed Aleynikov stole high-frequency trading code. The programmer countered that he was simply trying to strip out open-source code that he had added and return it to the community. Aleynikov spent a year in prison before he was exonerated twice.

Then, last year, Goldman actually opened up its Java library, GS Collections, making it available on open source. However, the firm was unable to take contributions from the outside world because it needed developers to sign a contributor’s agreement to protect the firm from being sued, Weir says. Goldman was eventually able to find a workaround by migrating GS Collections to the Eclipse Foundation and letting them accept contributions, but the issues around large firms contributing to open source remain.

“I think for the commercial organizations, particularly in finance, if you’re a little contributor on GitHub you’re unlikely to get sued; if you’re a large organization, you still are,” Weir says. “There is not very much case law in this place. No one wants to be the first to create one.”

Northern Trust’s Hardy, whose firm is not currently contributing code to open source, echoes similar sentiments. 

“Some of that stuff is being reviewed by legal, risk and compliance to figure out what the landscape is like,” Hardy says. “But I think every­thing you mentioned in terms of the complexities around that, I think all of that is valid.”

Licensing is a bugaboo for firms simply looking to use open-source software. Hardy says there are fewer than eight licenses—Apache, MIT and BSD among them—of which his team has the right to approve or deny usage. All other licenses are handed over to Northern Trust’s procurement and legal divisions for a decision. 

The licensing issue will continue to be a hurdle for any large financial firm looking to work with open source. It appears, though, that progress is coming, albeit slowly.

“It is glacially slow, unfortunately, but it is changing, I guess, with the changing of the guards,” says Paul Chou, CEO of LedgerX. “The much younger programmers who are coming to these firms grew up in a culture of open source. We are starting to see, sometimes, financial institutions now contributing to open source, which is something I don’t think would have happened 10 years ago.” 

Salient Points

• Open source is quickly becoming the place to find innovation, something that firms in financial services have slowly begun to recognize.

• One of the most important aspects of using open source is selecting the right community. While there is no simple solution to this problem, firms should look at the core contributors of the project, checking out their commitment to past ventures in the open-source space.

•  For a successful open-source implementation, a firm should be committed to the software and not view it as disposable because it’s free. 

•  Going forward, licensing will be a key issue on whether or not larger financial firms are willing to contribute back into open source. 

Below is an outline of some of the open-source projects currently being worked on at three of the firms interviewed for the feature.

BNY Mellon

John Weir, head of BNY Mellon's application platform and services (APS) group, sees four major categories where open source is trending in financial services.

The first is the user-interface space. Weir says most web applications are written in JavaScript, and there is a huge community with constant, fast-moving development in the space.

Big data is also a popular space for open source, Weir says, with Apache Hadoop driving a good portion of the work done in an area traditionally owned by vendors. Weir specifically cited strong use of the R programming language among data scientists, adding that there is a huge community of university professors maintaining and taking that library forward extensively.

The middleware space, which is Weir's focus at the New York-based bank, is also a prime candidate for open source. Weir says he sees it most present across infrastructure-as-a-service and platforms-as-a-service. Specifically, Weir cites Docker (containerization application), Apache Mesos (cluster management) and Kubernetes (automated deployment, operations and scaling of containerized applications) as areas of interest.

Finally, Weir points to the actual code libraries-JavaScript and Python-as areas that continue to see growth and activity.

BNY Mellon is hoping to take some of the things it has built and move them into open source. The issue-as is the case with many large financial firms looking to do the same-is licensing. "The challenges there are really still mostly corporate legal," Weir says. "At the end of the day, you're interested in making sure that you haven't violated anyone's IP rights."

LedgerX

Zach Dexter, CTO at LedgerX, is well-versed in the open-source space as the bitcoin exchange and clearinghouse was built from scratch using only open-source and in-house code.

Unsurprisingly, Dexter's focus in the open-source community is a major concern of his firm's day-to-day operations: network communications. According to Dexter, connectivity methods to exchanges, like IBM's MQ, are outdated.

LedgerX has an open-source protocol with its front-end software called efficient financial messaging (EFM) that was developed internally and released on GitHub. By offering EFM on open source, users can modify the software used to connect to LedgerX.

"If you're a participant of LedgerX and you want to change the way you handle incoming messages and send trades, no problem. You can make that change using one developer on your own serves and then submit that patch back upstream in a process called continuous integration," Dexter says. "So that is a big plus for us: The ability of our customers to use continuous integration to constantly be integrating changes to our software into the master copy of that software so that other people can benefit from it."

Northern Trust

There is a bit of excitement in Len Hardy's voice as he rattles off the different projects the Chicago-based asset manager is working on in the open-source space.

The first item Northern Trust's director of architecture and innovation brings up is the firm's enterprise data platform (EDP). The core of the EDP is a data reservoir that is built on the Cloudera Hadoop platform, helping the EDP to become the delivery vehicle for all reporting and data to corporate clients.

Hardy says the firm also has eyes on the development environment space. Apache Tomcat, an open-source application service, and Docker are two specific projects he cites.

Pandora, a real-time monitoring tool overseeing applications and infrastructure that gives early warnings when there might be potential issues all via dashboard, has already been implemented, according to Hardy.