Financial institutions have historically been wary of open-source technologies. These firms are closed systems, valuing secrecy and information security as a means of gaining an edge over the competition. Six or seven years ago, however, financial firms began to recognize that a lot of the innovation coming out of Silicon Valley was in the open-source world, and began to cautiously move toward open-source engineering.

One example of a financial institution openly embracing this movement is Canada-based bank BMO Financial Group, which is using open-source technology to build a proprietary platform for calculating the credit risk of its loan books.

Sriram Rajaram, CIO for credit and operational risk modeling and analytics at BMO, says the impetus for the shift came from the business users—in this case, the modeling team, who use risk analytics to run forward-looking scenarios on its loan portfolios.

Rajaram says that post-crisis accounting and risk standards like current expected credit loss (CECL) models and IFRS 9 require banks to aggregate data from their businesses and feed that into simulations, testing the robustness of their loan portfolios in hypothetical situations that simulate macroeconomic events.

When these standards first emerged, around six or seven years ago, BMO opted for a third-party solution to perform simulation exercises. However, after a few years, the bank decided it wasn’t getting the elasticity and efficiency it needed out of this vendor tool.

BMO needs to be able to scale up its simulation exercises practically indefinitely, in part because as the bank grows, whether by acquisition or organically, its portfolio of loans also grows.

And the bank must constantly fine-tune its models, as regulators are always updating their specific applications of international standards for calculating credit losses. The EU, for example, issued a new set of principles in June on the treatment of Covid in IFRS 9 accounting models.

“Our regulators are asking us to forward-look our books for their entire lifetime across different macroeconomic scenarios that could happen. There’s a range of scenarios, and multiple ways those could go. Each gives a view of how your loss profile could evolve, depending on economic conditions,” Rajaram says. “There is a predicted path, and then there is a variation where you have to say, What happens if GDP takes a dive to -30%, like it did when the pandemic happened? What if it was -50%?”

With the imperative to expand scenarios over decades and along many possibilities, data volumes explode, Rajaram says.

“Think about what we are trying to predict: How is loan behavior going to look over 10, 20, 30 years? These regulations want you to look at the lifetime losses of the whole book—thinking about every loan, going through scenario-based evaluations for 30 years, and then calculating a present value of those loss evaluations,” he says.

The vendor solution was also costing more money than BMO had expected. Meanwhile, BMO’s modeling team was reading about the potential for performing analytics on Amazon Web Services’ (AWS’s) cloud, and pushed for Rajaram to explore these capabilities.

“The upgrades [to the vendor solution] were very expensive, we had ever-growing licensing costs, the simulation runtimes were anywhere between 11 and 14 hours long for an IFRS 9 simulation,” Rajaram says. “That constrained us a lot in terms of how much we could simulate to help management with different scenarios and analysis. Combined with the growing complexity of modeling frameworks and compressed quarterly reporting cycles, that set the stage for us to look at alternatives.”

Rajaram’s team looked to Python-based web development framework Django and the big data processing framework Spark, which is available under an Apache license. Spark is considered one of the world’s best and fastest open-source big data processing frameworks, and is the first choice for many businesses that want to perform machine-learning or analytics tasks on big datasets.

Spark formed the basis for BMO’s prototype platform, which would be running a custom methodology for forecasting losses built by the modeling team. Initially, Rajaram’s technologists did not see much benefit from Spark, and only saw a negligible improvement in runtime compared to the vendor platform. But they soon realized the problem: They were feeding it datasets that were too small. Initially, as they had just wanted to explore the tech, they had been working with small loan books. Spark’s benefits really come into play with big data, Rajaram says, and once they gave it full loan books, the prototype platform could demonstrate a 7x to 8x improvement in speed.

The prototype was then evolved into BMO’s proprietary platform—dubbed Cheetah—which Rajaram describes as a fit-for-purpose execution engine that runs the customized modeling framework, which is fine-tuned for a certain class of loan forecasting scenarios. This methodology allowed BMO to configure Cheetah for its own needs, rather than relying on the vendor solution to adapt to the bank’s models.

BMO is still running Cheetah on-premises, but the platform was developed to be cloud-native, and can run on AWS or Microsoft Azure (though Rajaram says AWS is better for these kinds of tools). However, the bank is starting to pivot software development workloads to the cloud, and Rajaram says he anticipates that full production capabilities will be in the cloud by 2025.

The data that is fed into Cheetah is stored in BMO’s data warehouse.

“We get snapshots for the process that is required: monthly or quarterly, depending on the model we are running. The simulation engines then abstract away the scenarios and modeling, so we can choose what scenarios to run and what model version,” Rajaram says. “The data is there, you’ve sent the scenario and the model configuration, and the execution engine takes care of the rest. It says, ‘OK, I know what model version and snapshot you’re running,’ and it produces the simulation and returns the results.”

Once Cheetah goes fully into the cloud, performance can be scaled as needed. “When we go to the cloud, we’ll be able to just ask, ‘How many scenarios do you want to run?’ There is absolutely no restriction! It’s infinitely scalable for that context,” Rajaram says.

Right now, in its on-premises operations, Cheetah can run three IFRS9 simulations concurrently at 45 minutes to an hour per simulation, and five CECL simulations concurrently at 15 to 25 minutes per simulation.

“Now we are somewhere around 10x to 11x, which we have achieved on the platform itself. Once we are in the cloud, I will be able to do three scenarios or 300—as many as I want—in 45 minutes,” Rajaram says.

Talent scout

Finding the talent that understands open source is a constant struggle for most firms. Rajaram says BMO already had the human capabilities in place to deliver Cheetah. Back in 2016, when the bank was using the legacy solution, the vendor planned its own migration to cloud and open-source, and BMO started a process of upskilling in Rajaram’s team.

BMO’s approach was to go after college talent at campuses near its offices, including the University of Toronto, University of North Carolina at Charlotte, and North Carolina State University (Rajaram himself is based in Concord, NC). The bank looked for academics who were aware of both modeling data and technology—people who combined engineering chops with knowledge of the use case. This turned out to be those who combined a computer science degree with something like economics, or quant finance experts, for example.

The recruits were shadowed by tech leads and given room to experiment without being pressured to deliver on model changes. As this talent pool evolved and as BMO moved away from the vendor solution, these recruits contributed to the development of Cheetah’s prototype, suggesting instances where open-source tech could work to improve the platform.

BMO also had to consider that its existing staff might not have had significant experience with open-source technologies. Rajaram’s team built a user interface that abstracted away much of the workload, and they changed as little as possible as Cheetah evolved.

“There are certain things we did not change: the data process, the outputs. We just changed the compute engine,” Rajaram says.

As the team finished the prototype, it started bringing the business on board to give staff a feel for the solution and how they would interact with a Spark-based framework.

“The business had a few experienced folks who came from non-banking industries. They quickly became our champions, who could explain what was happening and how to interact with the solution,” he says.

Though BMO’s platform is a version or two behind the open-source community, Rajaram says the bank doesn’t need to be exactly at the latest version to reap benefits.

“That still gives us a lot of advancements, and two versions in open source is equivalent to six months. In the traditional vendor space, two versions are more like several years away,” he adds.

Nicholas Kolba, CEO of Connectifi, a startup tech provider focused on application interoperability in financial services, has pioneered large open-source projects including Chromium and the Dojo library, and led the creation of the FDC3 protocol, a set of coded specs that enables application interoperability, under the auspices of Finos, where he previously served as a board member. Kolba has seen firsthand through his work and his involvement in Finos how a “sea change” has occurred in banks’ attitudes to open source.

Kolba says that BMO’s approach of targeting people who understand not only programming but also the use case, is a good one, as these engineers are often the most effective, and have a more holistic approach.

“That’s been one of the most interesting developments in software over the past years: With the kind of engineering that happens in open-source ecosystems today, people are able to work at different levels—engineers can work in the application space, in the space of the actual use case,” Kolba says.

Open embrace

Banks have genuinely embraced open source, Kolba says. Most broadly, they have recognized that they need to position themselves as tech companies to stay competitive and to attract and retain the best engineers.

And they’re not just consuming open-source tools; they’re also contributing to them, Kolba says. Banks are starting to recognize that being involved in open-source projects gives them influence over the development of tools used throughout the business. Finos has seen significant contributions from banks, he adds. For example, Goldman Sachs has contributed an internally developed data management platform, Legend, to Finos, while JP Morgan contributed Perspective, an interactive data visualization tool.

“Banks are realizing that everyone is dependent on the open-source ecosystem, and benefiting a great deal from it. And if they’re not contributing to it, there’s a real risk that the tools they rely on—from Linux up to things like FDC3—won’t be around in the future because no one will keep maintaining it for free if there isn’t support,” Kolba says.

He adds, however, that banks’ contributions to open-source communities must mature. “It’s great that banks are participating more in open source and are present, but often they’re present kind of as cheerleaders only. They’re not necessarily driving these projects. Many contributors are just like, OK, I’m going to throw this project into open source so I can put this on my resume, and then I’m going to move on to another bank and do something else. And those projects just sit there and die.”

There isn’t much incentive in many organizations to prioritize open-source engineering, with engineers encouraged to become maintainers of projects. “Nobody’s telling them their bonuses are dependent on their participation in the maintenance of this open-source project,” Kolba says.

Without maintainers, projects wither on the vine. Kolba points to Microsoft’s purchase of npm, a major JavaScript developer platform that was getting so big, developers worried that open-source maintainers would not be able to keep its registry robust.

“What would have happened to npm had Microsoft not bought it? We’re taking it for granted that someone will pick up the slack. It would be great to see some banks jump in there, be more proactive and actively participate in the projects they rely on most,” Kolba says.