State of the tech: A half-year check-in with large language models

​I’m a born-and-bred New Yorker. I grew up in Battery Park City in Lower Manhattan, learned to take the subway at a young age and was raised a die-hard Yankees baseball fan. Part of that NYC upbringing also included frequent trips to one of my favorite places in the city: the American Museum of Natural History on the Upper West Side.

I spent a lot of time going to the museum to learn about indigenous tribes, particularly the ones that occupied the Northeast or Eastern Woodlands. There are also halls dedicated to indigenous tribes of the Great Plains and Pacific Northwest. The collections include carvings and paintings on materials like wood and cloth, and they typically tell stories, both fact and fiction. One could even say the carvings are early examples of data.

In the present day, hearing the word “data” may evoke images of numbers on a screen, inputs in a Microsoft Excel spreadsheet, or pages of a report. Yet data has existed for as long as people have. It’s well-documented that carvings and paintings were used to keep track of daily life, and serve as evidence that humans have been collecting and storing information for thousands of years.

What has evolved is how we interact and work with that information—for example, moving from cave walls, to parchment, to typewriters, to computers. Data has become digitized, and we now look to tools like the cloud to help us manage it and make access to it more seamless. As technology evolves, we can expect even better tools to make data access easier and more productive.

When a new technology is introduced to the capital markets, the industry gets excited and tries to find a way to add it to their toolbox, sometimes before it’s been determined that there is an applicable use case (staring directly at you, blockchain).

Right now, all eyes are on large language models (LLMs), which underpin the generative AI efforts that we’re currently seeing today. I’ve been paying close attention to this space for the last six months, and I’ve noticed that the products, announcements and strategies laid out seem to be tackling data and how to deliver more information, faster.

Familiar territory

Technologies that incorporate machine learning aren’t new to capital markets. Machine learning has found several use cases across various asset classes. Here are just some of the projects that took shape last year.

“We’ve been using machine learning for a very long time; we haven’t been using a lot of large language models—that neural network portion of it,” says Frank Tarsillo, CTO of S&P Global Market Intelligence. “When we look at the industry today, specifically in the use cases we’re seeing at S&P and others, you get this use of natural language processing (NLP) to search and discover information to be able to provide text summarization or sub summaries of information back in natural language form that a human being can consume.”

Similarly, Gideon Mann, Bloomberg’s head of machine learning product and research, told WatersTechnology back in March that LLMs could build on what NLP has already done. “The interesting thing about the LLM technology is that, out of the box, after you’ve trained on all of these datasets, it gives very good performance on tasks like named-entity recognition,” he said. “So part of it is the things we were already doing with NLP, we’ll be able to do better, faster, and cheaper using LLMs. And then there are probably things we could never do because the cost of setting up an engineering team and doing an annotation exercise was just too large.”

Bloomberg became the first financial data provider this year to release its own LLM, BloombergGPT. The data giant is looking to have BloombergGPT assist—and provide more depth to—the Terminal’s sentiment analysis, named-entity recognition, news classification, charting and question-answering capabilities, among other functions.

And last month, Moody’s announced a partnership with Microsoft focused on generative AI that will look to enhance its current offerings, one of which is through Moody’s CoPilot, an internal tool that will combine proprietary data and research under Moody’s Analytics with LLMs and AI technology that sits under Microsoft.

Sergio Gago Huerta, managing director of AI and machine learning at Moody’s Analytics, says the platform will offer a secure way to leverage LLMs. Posing questions related to financial markets to ChatGPT would currently be difficult due to the chatbot’s lack of knowledge prior to 2021 and the fact that it can “hallucinate” false information. “That’s exactly why we call these things co-pilots and not pilots. It’s critical that there’s a human in the loop, validating and verifying the content,” Huerta says.

Having verifiable data is key to preventing—or at least identifying—those hallucinations. “If you ask a system, ‘What is the latest rating of company X? What are the risk factors? Is there anything that happened in the news five minutes ago that could be impacted by that?’ we’re able to pull that together and give you exactly the source or citation that makes that claim, which is impossible with an out-of-the-box LLM,” he says.

Steve Rubinow, former chief innovation and technology officer at the New York Stock Exchange and lecturer at the Jarvis College of Computing and Digital Media at DePaul University, says domain specificity with these models will pay off productivity-wise. “If you understand the domain, this could be a very useful productivity tool for you. And if it’s your own data and it’s your own model, whether it’s something you’ve purchased or you’ve acquired from open source, it’s great because you have full control over it,” he says.

In April, I wrote that data providers were the best candidates to make use of this technology due to the data at their disposal. Others agreed, and it appears we are making more moves in that direction. S&P’s Tarsillo says the data vendor is looking at domain-specific LLMs. In May, Morningstar released Mo, its AI chatbot designed to deliver insights in a conversational format. Morningstar is utilizing Microsoft Azure’s OpenAI Service for the chatbot and it’s just the first offering rolled out for what will be built on the Morningstar Intelligence Engine platform, which utilizes the provider’s equity and managed investment research, editorial, and ratings.

The tool chest

Over the past few months when I’ve spoken to vendors utilizing LLMs, many of them have looked to the Microsoft Azure OpenAI Service for their offerings. When I ask why they didn’t build one themselves, most cite the large amount of compute power needed or the lack of data available if they aren’t a data provider. Some balked at the idea and called it “insane.”

I understand that reaction. This stuff isn’t easy. “LLMs are hard to scale and they’re really, really expensive,” a machine-learning engineer at a financial data and technology provider told WatersTechnology. “And you’re basically boiling the ocean to make these models—[there are] trillions of parameters in the model. There are all sorts of custom hardware being built to run them right now. But they do represent a significant leap in technology.”

History doesn’t repeat itself, but it often rhymes. And we’ve been somewhere like here before—with cloud. In the not-too-distant past, financial firms embarking on their cloud journeys largely opted to leverage private clouds. That slowly gave way to hybrid private–public models. And now here we are, in a time when some are looking to sunset their datacenters altogether. AI may follow a similar roadmap.

Some will choose to do things internally. Bloomberg’s LLM shows what you can pull off when you aren’t boiling the ocean and don’t have, or need, a trillion parameters. But on the flip side, Moody’s strategy also shows the practicality of having a partner in AI. Microsoft’s bet on OpenAI may prove to be one of its best decisions yet.

Despite the differences in opinion on how to approach the build versus API connect strategy, it’s clear that people can agree on two things: First, data is the most important piece of this tech, and second, LLMs benefit from the tech that came before it.

S&P’s Tarsillo says data remains critical to the value LLMs provide those working with them. “If you don’t have quality data—connected, linked, curated, disambiguated content—you’ll get less of an outcome and you’ll get more hallucinations out of the technology,” Tarsillo says. WatersTechnology’s editor-in-chief, Anthony Malakian, wrote a few weeks ago that good data management and a good foundation will help firms go far with this tech.

And big data, cloud, and open-source have set the foundation for what large language models can build upon. What has come before it has had staying power, and I don’t think it’s unfair to say LLMs will see the same. Or as Tarsillo put it to me: “Everything has a place.”

Those early cave drawings were, in part, meant to inform others of potential dangers and predators lying in wait ahead. Data has always driven trading—sometimes driving up the price of a tulip to astronomical heights. Good data and outputs matter.

But what’s clear is the industry has evolved from the need to have data, to the era of big data, to the need to provide context and analytics around that ever-expanding sea of information becoming ever more paramount.

LLMs represent the next leap in innovation, and while it’s still early days, what we’re seeing today with GPT-3 and GPT-4 will—in the near future—be viewed as primitive as, well, a cave drawing. Are you and your tech teams prepared for that kind of rapid evolution?