Hidden danger: As AI permeates finance, cybersecurity moves to the forefront

Daryan Dehghanpisheh has been a busy man lately. This year has marked a significant shift in the world of technology, as people can’t seem to get enough of generative AI and the large language models (LLMs) that underpin it. Dehghanpisheh and the company that he co-founded, Protect AI, have been in conversation with some of the largest sell-side firms and data providers about the security of these models. It’s an area in the AI/ML field that he feels deserves as much consideration as model bias.

“We’re running so fast on innovation—we want to consume it. The last time C-suites were this excited was with the introduction of the App Store in 2008, where everybody was like, ‘We need an app,’” says Dehghanpisheh, whose friends and colleagues simply call him “D”.

“Today, it’s, ‘We need an AI thing; we need a chat thing.’ So, the hype has just gotten in front of basic, foundational building-block components.”

Innovation is indeed moving at a rapid pace. Since ChatGPT’s commercial debut last November, AI has become a major talking point across capital markets. Finance-specific models like BloombergGPT or LTX’s BondGPT have also demonstrated the way this technology can be utilized across financial services and the potential for AI to deliver enhanced workflows to users.

Amid that innovation, firms like Protect AI are looking to provide the tools and guidance for firms to adopt machine learning security operations (MLSecOps), where security sits at the heart of AI development.

Attack surfaces

Dehghanpisheh knows a thing or two about deploying tech strategies. Prior to co-founding Protect AI alongside former Amazon Web Services and Oracle AI engineers, he led AWS’s global AI/ML solutions specialist architecture team, working with the financial services segment. He was also previously the senior vice president and managing director of technology and data products at the New York Stock Exchange.

To highlight the security dangers that lurk in the shadows of machine learning’s evolution, he points to MLflow. The popular open-source platform is used for managing the end-to-end machine-learning development lifecycle. It allows ML engineers to experiment with, manage and deploy different models written in different code. It can also enable models to be reproducible, so they can be shared with others.

In March, Protect AI discovered a major vulnerability in the platform that could allow a remote, unauthenticated bad actor to acquire the models being run in the MLflow server, secure shell (SSH) keys and even cloud credentials.

“What’s interesting about MLflow is that it is such a popular, ubiquitous, and important tool that is used in almost every single financial institution, in some form or fashion,” Dehghanpisheh says. “And yet, I can take over your system, your cloud account, I can steal all the models, all the model registry, all the model intellectual property, and any of the datasets that traverse that.”

Protect AI informed MLflow maintainers of the vulnerability and it was patched in the 2.2.2 version of the platform. MLflow maintainers did not return WatersTechnology’s request for comment.

The reason for the platform’s popularity, especially among financial services firms, is because, as a machine learning engineer at a large trading technology provider puts it, smaller firms can experiment with ML models for analytics without having to hire tons of engineers, and it allows larger firms to manage the cacophony of AI projects being undertaken across their organization.

For example, MLflow has built-in tools for “versioning” a model and tracking its performance. Let’s say you have three engineers working on three different projects developing models. One engineer wrote a model in TensorFlow, another wrote a model in PyTorch, and the third wrote one in Dax. Regardless of how they wrote those models, the model binary—or model artifact—can be “wrapped” in MLflow and deployed to, say, Amazon SageMaker or Databricks, and it will have a cross-platform API and wrapper.

The engineer, who has used MLflow in the past but not recently, says that if a firm is using multiple different frameworks—which is common in the capital markets because of the different needs of a trading firm or vendor to cover multiple jurisdictions, asset classes and services—then there’s “a whole bunch” of additional engineering work that the various teams do to get the end model out into production.

“If you’re using 20 different frameworks on your team for different domain-specific purposes, to deploy that is a freaking nightmare,” says the engineer, so MLflow can help streamline these stages of development and deployment.

But problems arise because non-engineers tend to use these wrappers a bit too liberally, and there often isn’t a coherent deployment strategy.

“The world of MLflow is the Wild West. It’s been the Wild West for more than 10 years now,” says the engineer. “The problem is that you take these people who are physics post-docs, math PhDs, comp sci PhDs—people who work in research—and by and large they’re not applied engineers. They’re not carpenters building houses; they’re people doing engineering research at a university and you’re throwing them on a job site. They understand a crazy amount about the theory and know how to put a stabilizer up, but inevitably the house won’t really be that secure.”

As a result, you have a lot of data scientists working in, say, Jupyter notebooks who are “scratching off Python code,” then bundling it up in a model and wrapping it in different frameworks that are themselves in rapid iterative development by other data scientists and researchers. From there, they push a “gi-giga-blob” out to production that has special run-time characteristics, which makes it more difficult for them to run alongside normal hardware and deployment patterns, which creates additional unforeseen vulnerabilities.

It’s a very nascent open-source technology being developed by a relatively small group of people. The security vulnerabilities are really not well known

Machine learning engineer at large trading tech vendor

“All these layers together contribute to a very volatile deployment. It could be MLflow today; it could be TensorFlow tomorrow; it could be some crazy code a data scientist shipped to prod after that,” says the engineer.

“It’s a very nascent open-source technology being developed by a relatively small group of people. The security vulnerabilities are really not well known. Some will treat it like any other web application, but at the end of the day there are a lot of very specific run-time properties of a machine learning model that make it different from just a Java service or a Python service running in the cloud.”

Project AI’s Dehghanpisheh posits that a further complication is that many financial services firms’ security systems don’t necessarily scan for machine learning supply chain assets—the necessary tools aren’t built into the systems.

“Even if they are, there tends not to be version control or version checking—the difference between version X and version X-plus-one,” he says. “And then on top of that, because of how machine learning models, code and systems are built, there’s no way for a security person to go and say, ‘Show me every model that has MLflow and every system that has MLflow 2.1 or prior.’ You can’t inventory it.” Protect AI, in its blog post detailing the vulnerability, did offer those using MLflow a tool that could be downloaded from GitHub that could be used to patch systems.

Attack surfaces

“AI/ML is a critical new technology domain. And like any technology domain, it is going to be susceptible to a mix of classic styles of attack and new styles of attack,” Dehghanpisheh says.

Setting different defense postures, as well as different methods of maintaining governance, risk and compliance terms, is a necessity. Incident response management policies are also key, just as they would be in cloud deployment. Looking back at the development of cloud in the capital markets, a simple way to think about it, Dehghanpisheh says, is that, at first, clouds were “largely on-prem and self-directed.” But as the Big Tech companies have taken root deeper in the capital markets, banks, asset managers, exchanges and vendors are all thinking about how they want to deploy their tools in the cloud.

“It wasn’t that things necessarily had to be different—you still needed ease-of-access controls and data security—but how you did that in a cloud environment or a hybrid cloud environment is materially different than how you were doing it when you had all of the technology under your own umbrella,” he says.

It requires you to start thinking about your platform in terms of models, not just infrastructure

Daryan Dehghanpisheh

Even before the arrival of ChatGPT, AI and ML were permeating capital markets across various use cases. According to Nvidia’s 2022 State of AI in Financial Services survey, respondents reported their highest investments in AI/ML were for transaction fraud detection, conversational AI, algorithmic trading and KYC/AML. But those investments are still small and limited, with the highest percentage in the Nvidia report being 31% of firms. A former chief data officer at a large bank tells WatersTechnology that the amount of data exposed to machine learning models is still relatively small: “Machine learning applications are far and few [between], and they are focused on very specific use cases.”

Nonetheless, Dehghanpisheh says the models need to be treated as unique threat surfaces. “It requires you to start thinking about your platform in terms of models, not just infrastructure,” he says. “I think the biggest shift that things like these large language models and others are creating is the concept of models as platforms. The platform before was, I’m debating over server type A or server type B, which became, okay, now I’m debating about cloud vendor one versus cloud vendor two. As a result, I’m now going to be writing applications that are built on foundational models.”

Here’s the rub: those foundational models can then be vulnerable to hacks. Methods like adversarial machine learning, where a bad actor may extract information about the attributes and characteristics of an ML model and then attempt to manipulate the data that goes into it to alter an outcome, have been the focus of recent research. But Dehghanpisheh says traditional hacking methods can also target the supply chain of software components and allow the theft of models and their underlying data, because they’re not secure.

Ultimately, it’s not just a technology issue, nor is it just a philosophical dev issue—it’s also a people problem, says the ML engineer. Monitoring what kind of development—and what kinds of open-source tools are being deployed throughout an organization—and finding the right talent to oversee these projects, is the balance that firms will struggle to find.

“To be fair, open source presents a unique opportunity to solve the problem in a better way. Closed source exacerbates the problem; with open source, you get more eyes on it,” says the engineer. “We know a lot more about computer systems today, but only a subset of that applies to the machine learning domain, and there’s so much momentum—so many bodies being thrown at it—that it’s tough to keep up with everything. It’s so hard to find machine learning engineers. People who have enough math knowledge and research knowledge and machine learning knowledge and engineering knowledge—they’re so rare.”