A problem shared is a problem halved—this is the thinking behind a group project led by Goldman Sachs and JP Morgan. The two banks are working together to mobilize industry working groups, where technologists and market data specialists can collaborate on how to deliver and consume market data via the cloud.

The first set of engineering sessions, steered by John Chappell, global head of market data at Goldman Sachs, and Alistair Kerr, global head of market data at JP Morgan, was held this week at the World Financial Information Conference (Wfic) in Prague.

As financial firms like Goldman and JP Morgan undergo individual transformation projects, the pair thought it would benefit key stakeholders in the industry to join forces and think more strategically about how all firms can best leverage the cloud. In doing that, they brought some of the brightest engineers in the business together in one room to share expertise, learn from past mistakes, and generate ideas to resolve common problems by using cloud technology.

“It’s about trying to create a community where engineers can share their war stories and ideas,” Nigel Phelan, market data services architect at JP Morgan, tells WatersTechnology. “A part of that is getting that constituency of people that use these [cloud] products together to talk about what works and what doesn’t work, and they can then independently feed that back to the vendors and say, ‘Here are the problems.’”

Although WatersTechnology was not present at the engineering sessions, we spoke to more than half a dozen attendees.

So everyone’s been used to driving around on back roads and alleyways, we’ve got loads of congestion, it’s really hard to read a map, and you often get lost—it’s really slow. But now, what we’re moving to is this super well-connected motorway.

Senior data executive

Sources say the sessions were attended by over 100 people, with roughly 50 participating in the virtual lab, from all parts of the industry—including banks, exchanges, and vendors. Amazon Web Services held a demo on how to query data in AWS Data Exchange (ADX) and Google Cloud Platform (GCP) also assisted by allowing the group to experiment with their virtual lab environments and stream data in the cloud from providers like FactSet, TP Icap’s Parameta, and the Chicago Mercantile Exchange (CME).

There are two main domains that the working group and the industry as a whole are trying to explore: data at rest and real-time data. Data at rest, such as reference data and static pricing, is highly suited for cloud computing environments like ADX, Snowflake, and Google BigQuery.

“Those tools are far superior to what you have on-premises. So it naturally makes sense [with data at rest] that you move the storage and the analytics that you want to build on top in the cloud,” says a senior executive and data consumer, who attended the sessions.

Distributing and consuming data directly from the cloud can remove multiple points of friction. For instance, today, a consumer might request data from their vendor, and that data would then be distributed from one of their FTP servers, which then would need to be parsed and reformatted so it’s compatible with end-user applications. Then the consumer would need to consistently check if that data is updated on their end.

“With Amazon’s Redshift and ADX, you can store it in one place for customers; you don’t have to create these continual instances. You have one copy of the data, and you’re able to project different views of that data depending on who the client is and what they’re permissioned for,” the senior executive says. “And when you make changes, the way that Redshift works is the customer will automatically be notified when there’s an update.”

Static data, however, isn’t the concern. The problem many members of the working group are trying to solve is using the cloud to access real-time data. This is where existing legacy on-premises systems become problematic because the way exchanges’ or data vendors’ systems are built is not easily reconstructed to be low-latency on the cloud.

“The second debate is all about real-time data and that isn’t a naturally good fit for cloud because people who are running real-time ticker plants will likely have relatively exotic on-premises setups that will be using certain hardware technology; they’ll be using dedicated networks and multicast. So there is a lot more complexity and a lot more concern around deterministic latency, where you don’t have these big outliers and jitter,” the senior executive says.

There are already some solutions being discussed on how to tackle the real-time challenge. The senior executive says cloud service providers (CSPs) are building new bespoke low-latency solutions and the ability to run processes across massive cloud provider estates could speed up the time to data.

For market data consumers, ideally all forms of at-rest and real-time data—from exchanges, terminals, alt data providers, and so on—will be accessible from a cloud-hosted environment.

“What you really want is to be able to see what data is out there and be able to select the relevant bits of it. And you want that to be across the whole universe of data vendors and see all of the content exposed in one place,” says Phelan, speaking about both the engineering sessions and internal JP Morgan solutions used for cataloging data.

A sign of the times

There are many reasons to believe the industry is moving in the direction of this new data distribution model. In the last 18 months, the industry has seen major multi-year deals emerge—notably the CME’s 10-year contract with Google Cloud and Nasdaq’s decision to move to AWS Outposts.

Other exchanges, like Deutsche Börse and the London Stock Exchange Group (LSEG), are also taking the leap by shifting operations to the cloud. In March of this year, Bloomberg announced it would offer its B-Pipe real-time market data via GCP.

Of course, this will require a lot of work from all stakeholders involved. And there could be big trade-offs with trying to use a multi-cloud data model because each cloud provider operates differently and offers varied bespoke tooling, which as of right now isn’t portable between different CSPs. And what sacrifices will consumers be willing to accept in a multi-cloud market data environment?

“If you want to use all of the extra value-adds that make you more productive on the platform, then the price of that is that you will be more coupled to the platform, and moving off of it will be more difficult, but the return you get is it will probably be much quicker for you to develop stuff and your time to market will improve. So you have to make a choice,” Phelan says.

Evangelists of the technology hope that if there’s a critical mass of data vendors and consumers on the cloud, this will create huge efficiencies by removing the need for circuits and shutting down old systems and datacenters. Additional, long-term, downstream benefits could also be a reduction in carbon emissions using smarter and less energy-intensive compute power.

“So everyone’s been used to driving around on back roads and alleyways, we’ve got loads of congestion, it’s really hard to read a map, and you often get lost—it’s really slow,” says the senior executive. “But now, what we’re moving to is this super well-connected motorway.”