Hacked off: banks demand answers after Ion cyber attack

Back-office staff at a bank in London arrived at work on the last day of January to find the software they used to process futures trades no longer worked. They promptly informed the vendor, Ion Group, and waited for a response. And waited.

“Hours after we reported the issue, we got an email confirming the system was down,” says an executive at the bank.

That morning, Ion discovered that some of its systems had been infected with the LockBit ransomware, which encrypts files, steals data and deletes backups. The affected servers were quickly disconnected, leaving 42 customers unable to access key services.

After confirming the outage, Ion told the bank in London that it was working to fix the issue and that the estimated recovery time was three days, with more updates to come.

There are a lot of really angry people out there, up to and including the regulators

Derivatives lawyer

Six days later, the software was still not back up and running and there had been no further word from Ion. “As of right now, I don’t know where we’re at,” the executive said on February 6, his frustration seemingly boiling over. “I’m sure they were dealing with a lot, but it’s not hard to send out an email.”

That sentiment is widely shared. “There are a lot of really angry people out there, up to and including the regulators,” says a derivatives lawyer who has had discussions with affected customers. “There was just a gaping black hole of information that left everyone absolutely gobsmacked.”

Clients were left to face the music. As the outage continued, regulators began asking increasingly pointed questions. They found few answers. “When something like this goes down, the regulators are calling up the clearing banks or broker/dealers,” says a source at a US clearing firm. “But really, they should be calling the technology company.”

The UK’s Prudential Regulation Authority and Financial Conduct Authority are investigating the cyber attack.

Risk.net and WatersTechnology spoke with more than 40 banks, brokers and industry sources, many of whom expressed deep dissatisfaction with Ion’s initial response to the attack. “Their comms was horrible,” says an exchange source.

Ion declined to comment for this article. However, a source close to the firm rejected the criticism of its response. “They did hold multiple Zoom calls every day. I think it started at about four calls a day and then moved to twice a day and [they] invited all qualified contacts. They communicated quite clearly with their customers on where exactly they were every day.”

Clients confirm calls were held, but that they were not as informative as some had hoped. An executive at a second bank says Ion routinely failed to follow-up with the updates and security briefings it promised and that its usual contacts at the company were “eerily silent”.

The cyber security measures taken by some clients added to the chaos and confusion. Several banks and exchanges blocked emails from Ion as a precautionary measure, cutting off a vital channel of communication. “When we heard about the hack, we closed anything coming from Ion,” says a clearing source at a European bank. “They could not send us email. They were telling us on the phone, ‘we have sent you emails with information’, and we told them, ‘we cannot receive your emails’.”

A futures industry source who has been working with affected banks says Ion’s communication improved after the first week. “By the end of the first week of Feb, I think they were providing more transparency into their recovery process and where things stood. They came around and got a bit more in front of it.”

Even so, clients say they were never given the full picture. More than a month after the hack, banks were still trying to get to the bottom of what happened. “As of last week [of March 6], the concern was that they haven’t received a forensic report yet to give comfort that there’s been a full investigation and that the entry point of the attacker was known,” the futures industry source says.

“People don’t have the full facts at this point,” confirms a clearing executive at a fourth bank that uses Ion’s services.

Ion has retained the cyber security firm CrowdStrike to conduct an investigation. CrowdStrike declined to comment. However, two cyber security experts say forensics reports are rarely shared with clients, unless required by contract. 

In the absence of a detailed explanation from Ion, clients have been left to speculate. Questions have been asked about the vendor’s cyber security practices, as well as its business strategy of rapidly acquiring rivals, stitching their systems together and cutting costs to drive profits. Three clients want those questions answered by an independent auditor. Two of those plus another bank have also asked their lawyers to look into whether Ion breached its service level agreements (SLAs) with them.

At least one client has already decided to “move away from Ion” following the cyber attack. Another would like to do the same but concedes that escaping Ion’s grip won’t be easy. “The problem is that Ion owns a lot of solutions where there is no competition,” says an executive at this firm.

Left in the dark

After receiving confirmation on January 31—a Tuesday—that Ion’s servers had been hacked, clients scrambled to activate their business continuity plans.

Firms that used Ion’s software for trade processing and matching—called XTP and XTP Clearing—were among the hardest hit and had to switch to manual workarounds or backup providers to process trades. Risk.net and WatersTechnology understand ABN Amro, HSBC, Macquarie, RBC and UBS were in this camp. Several other banks, including some large dealers, used Ion for non-core services, such as data and record-keeping (see table below). CME said on March 8 that one-fifth of its 67 clearing members were affected by the service outage.

A few of these firms had backup providers that kicked-in, mitigating some of the disruptions. Others had to resort to manual processes.

At the bank in London, which uses XTP Clearing, back-office staff worked through the night that week, directly logging into exchange websites to clear trades. The business was “able to keep operating”, the executive says, but the service outage “had a massive, massive impact” on the cost of doing business.

The day after the hack, clients joined a conference call to learn more about the incident. It went badly. On the call, Ion’s founder and chief executive Andrea Pignataro started “screaming” and “yelling out at his staffers”, according to three people familiar with the incident. The outburst “shocked the clients”, one of these people says.

Users of XTP Clearing—a product formerly known as Seals—were initially given an estimated recovery time of three days. That target was missed and at least one client claims it received no further updates. As the weekend approached, it was still unclear exactly how long the outage would last.

By now, banks were growing increasingly concerned about the hackers’ threat to expose confidential client data if a ransom wasn’t paid by Saturday, February 4. Hours before the deadline, the hackers told Reuters their demands had been met by “a very rich unknown philanthropist”. Clients sought confirmation but only found more confusion. On calls that weekend, representatives of Ion variously denied they paid the ransom or “dodged the question”, according to two sources familiar with the conversations.

The notion that you’d be getting zero data from them was a little inconceivable. And now it’s conceivable.

Source at a US clearing firm

Finally, there was some good news. Clients were informed on February 5 that the restoration of the XTP system was around 80–90% complete and that at least two clients would have their services re-activated the following day. But at this point Ion had shared only limited information about how it was compromised and appeared to be leaning heavily on CrowdStrike to reassure customers that it was safe to reconnect to its servers.

That wasn’t enough for one bank’s internal cyber security team, which wanted to perform its own due diligence before reconnecting. The bank was still conducting security checks and had not re-activated its services three weeks after the hack.

Some contrast Ion’s response with that of market infrastructures. After blocking network connections with Ion to prevent contagion, exchanges and clearing houses extended end-of-day processing deadlines and made staff available to manually retrieve and send files where necessary.

A source at a US clearing firm says Ion was unable to provide it with any data after locking down its systems. “The notion that you’d be getting zero data from them was a little inconceivable. And now it’s conceivable,” they say. “Firms are asking from a business continuity planning perspective, how do we cover ourselves going forward? That’s been a lot of the talk on the Street.”

On the face of it, Ion’s response and recovery times fell well short of industry standards. A product head at a firm that provides data services to a similar client base says its SLAs stipulate that clients must be notified of systems failures within 15 minutes, and any issues resolved within 24 hours. The firm is also obligated to provide a final report on the causes and cures for any outages within five business days.

The product head concedes these timelines may be “thrown out of the window” following a crippling cyber attack, which may result in staff being locked out of critical systems and tools needed to respond and restore services. “There may be mitigating circumstances, if they can’t log into their IT service management tool to notify clients,” he says. “Maybe they haven’t been transparent because maybe this isn’t truly over.”

Some clients also acknowledge that Ion may have been limited in what it could share when it was still getting to the bottom of the hack and dealing with the ransom demand.

But that only goes so far. “What happens during resolution should stay in-house—the team needs to focus on the fix,” says the product head. “However, once it’s up, there should be a transparent report.”

How did LockBit do it?

There are still plenty of unanswered questions, starting with how Ion’s servers became infected with LockBit’s ransomware in the first place.

“We don’t know how they got in,” says Yiannis Pavlosoglou, founder of Kiberna, a cyber security consultancy, and a former UK chief information security officer at UBS. “It could have been phishing, spam emails, any other vulnerable areas.”

The futures industry source says representatives of Ion have verbally communicated “different things” to various clients about how the hackers got into its IT infrastructure. “To one firm, they even suggested that the threat actor got in using the compromised credentials of an Ion client.”

A risk manager at another firm says they heard from a reliable source that the hackers obtained the credentials of an Ion employee through a so-called spear-phishing attack, where emails that appear to be legitimate are sent to a targeted individual.

A basic defense against such breaches is multi-factor authentication (MFA), which prevents stolen or compromised credentials from being used to access networks. However, the futures industry source says one client confessed that they were not previously using MFA to log into their Ion-hosted services, although they are now. Since then, “a couple of firms shared that Ion was rolling out two-factor authentication as a remediation step post-incident”, this person adds.

Ion declined to comment on whether its clients and employees were required to use MFA before the cyber attack.

Oliver Tavakoli, chief technology officer at cyber security firm Vectra, says it would be bizarre if they weren’t: “Having a sizable part of your footprint not have MFA for end-users is at this point not really conscionable.”

If client credentials were used, that raises questions about how the hackers got into Ion’s back-end systems. “If it was through an end-user, was there some lack of logical segmentation between ‘user land’ and their production environment?” asks Richard Seiersen, chief risk officer at cyber firm Resilience.

Once in the network, the hackers would have scouted for vulnerable systems that could be encrypted with LockBit’s ransomware. A note Ion sent to clients soon after the hack—seen by Risk.net and WatersTechnology—cited an issue “involving VMware servers”. These are used to run virtual machines that can be accessed remotely and are a common target for hackers.

Around the same time, there were reports of a massive ransomware campaign targeting a two-year-old vulnerability in VMware’s ESXi software.

A senior executive at a brokerage firm says his contact at Ion confirmed that the hackers exploited a vulnerability in ESXi.

VMware publishes several security advisories on its website each month with accompanying patches to fix reported vulnerabilities.

In a blog post last year, the company specifically warned of an increase in ransomware targeting its enterprise class ESXi servers and cited LockBit among the key threats. A subsequent security advisory issued on December 13 urged users to apply a patch to a ‘critical’ vulnerability in ESXi. “A malicious actor with local administrative privileges on a virtual machine may exploit this issue to execute code,” VMware warned at the time.

On January 31—the same day Ion discovered it had been hacked—VMware issued another critical security advisory related to a vulnerability in its log-keeping tool that could expose users to remote code execution by “an unauthenticated, malicious actor”.

You can kind of read into that that CrowdStrike deemed whatever endpoint protection they were running to be insufficient

Futures industry source

While patches are available for these and other VMware exploits, IT departments don’t always apply them immediately. This is because patching servers often requires downtime and can be disruptive if done too quickly or without proper planning. They may instead opt to closely monitor vulnerable systems for unauthorized access—especially with VMware servers, which are typically not internet-facing and seen as low risk from a cyber security perspective.

The futures industry source says Ion has told clients that it is deploying a new endpoint detection system provided by CrowdStrike as part of its response to the cyber attack. “You can kind of read into that that CrowdStrike deemed whatever endpoint protection they were running to be insufficient,” this person says.

The length of the service outage, lasting more than a week for most customers, has also raised questions about Ion’s backup strategy and business continuity planning. “If their database was encrypted, what about their backup strategy?” asks Seiersen at Resilience. “Were they doing a 3-2-1 oriented backup strategy, were all those things put in place?”

The 3-2-1 backup strategy calls for maintaining three copies of data, with two stored locally on different systems, and one off-site.

An exchange source, though, says the fact Ion was able to rebuild its databases suggests it had backups and that the company deserves credit for its recovery efforts.

Buying trouble

The cyber attack has also heightened long-simmering concerns about Ion’s business model, which resembles a leverage buyout vehicle—acquisitions have left the company with around $6 billion of debt, according to Bloomberg—as much as a financial technology company.

After the hack, the futures industry source asked a cybersecurity consultant to provide an external risk assessment of Ion. “What he saw was indicative of a company that had acquired many disparate companies and had different technology teams using different technology stacks that were all under one Ion domain. He found a diverse number of different firewalls, different manufacturers of VPN concentrators, of different vintages.” The consultant concluded that Ion had “a large and diverse attack surface” due to its history of acquisitions “which naturally is difficult for a team to manage on a global basis”.

The clearing source at the large European bank also believes Ion’s business model played “a role”, adding: “A lot of those companies are growing like that—it’s not an isolated business model.”

Founded in 1999 by Pignataro, a former Salomon Brothers bond trader, Ion initially found success with a software package for electronic fixed-income trading, called MarketView. After selling a minority stake to private equity group TA Associates in 2004, Pignataro became a serial acquirer (see timeline below). In all, Ion has bought out around 30 financial technology vendors of all shapes and sizes, including suppliers of trading, risk management and back-office processing systems as well as data and analytics for all major asset classes.

Cyber experts say one of the big challenges for technology companies that make acquisitions—let alone this many—is properly integrating the systems they buy and bringing them up to the same security standard. “If they weren’t meeting expected standards, the question really becomes was it caused by some form of moral hazard, like cutting costs?” says Resilience’s Seiersen.

Ion’s push into cleared derivatives began with the purchase in 2008 of Rolfe & Nolan and its suite of back-office software, including the RANsys trade processing system for futures and options and a margin calculation engine, Margin Direct. The deal for Patsystems in 2012 gave Ion a front-end trading system for listed derivatives. It then added more back-office capabilities through the acquisition in 2013 of FFastfill, known for its Seals order matching engine and a derivatives trade processing system called Eclipse.

Around 2015, Rolfe & Nolan’s back-office solutions were rebranded as XTP and Seals became known as XTP Clearing.

But Ion wasn’t done yet. After private equity giant Carlyle became an investor in 2016, adding $400 million to Ion’s war chest, it began going after bigger targets, buying Fidessa for $1.5 billion in 2018 and Broadway Technology in 2020 for an undisclosed sum—although that deal was partially blocked by UK competition authorities. Ion also continued to expand its cleared derivatives capabilities through the acquisition of firms such as Clarus Financial Technologies, a provider of data and analytics.

Along the way, Ion gained a reputation for aggressively cutting costs and shedding staff at the companies it buys. Around a quarter of Fidessa’s employees—roughly 400 people—are reported to have left the business in the 12 months after it was acquired by Ion.

Last November, Ion’s full suite of cleared derivatives products was rebranded under the XTP name, which was previously only used for back-office services. The Janus Risk Manager and GTP front-office products were renamed XTP Risk Janus and XTP Execution, while the Charm system acquired from Clarus the year before became XTP Analytics. The XTP branded products were pulled together to create a fully hosted ‘end-to-end’ platform, called XTP Front-to-Back Suite. Less than three months on, many of these products went offline due to the cyber attack.

Stick or twist

Ion’s services were restored by the end of February, but some banks that were affected by the attack are now looking for other suppliers.

“There has always been interest to not depend on Ion because of the long, all-or-none contracts that you can never downsize, and the inability to only use some components,” says a source at a large bank.

An immediate question for banks is whether regulators will fault them for doing business with Ion if its cyber security was not up to snuff. The UK FCA, which is investigating the hack, has made it clear that it will hold banks responsible for ensuring the resilience of critical services, regardless of whether they rely on software vendors to provide those services.

“This incident confirms why we require all regulated firms to have appropriate operational resilience plans in place, including for when third-party providers are subject to outages or issues,” an FCA spokesperson told Risk.net on February 22.

Michael Ruck, a partner at law firm K&L Gates, says: “UK regulators are likely to look closely at the actions of the banks and whether they have met their own regulatory obligations, in particular but not limited to, conducting regular and appropriate risk assessments and audits of the software providers.”

Banks have asked lawyers to review their SLAs with Ion and determine if they were breached. At least one bank has resolved to end its contracts with Ion if that is the case. The futures industry source says other clients are considering ditching Ion’s hosted services and running its software on their own systems to reduce cyber security and operational risk.

Most large clearing banks have already replaced Ion with software built in-house. That isn’t an option for the mid-tier firms that make up most of Ion’s client base. “The vendors are charging on a flow basis, so under a certain volume, you’re probably better off with software-as-a-service,” says the head of clearing and prime brokerage at a European bank. “If you start to be significant, and you want to have more control over your activity, then you need to start thinking about having it done internally.”

In 2019, a group of European and UK banks looked at jointly building their own software to replace Ion. The plan, dubbed Project Cohesion, fell apart after they failed to agree on design and funding. “They never got traction because there were too many banks. All of them have very idiosyncratic means and no-one wanted to put up all the cash that it required,” says the source at the large bank, who previously worked at one of the firms involved in Project Cohesion.

Ion’s biggest competitor for clearing software is FIS, which is also favored by some larger banks that use third-party software. Other alternatives include firms such as Valantic and TransFICC, which provide connectivity to clients and exchanges, but not the complete front-to-back solutions that Ion and FIS can.

“They may not be full Ion replacements, but they do [provide] some of the components and let the banks have intellectual property and have their quants do some of the work themselves,” says a clearing source at a second large US bank.

The executive at the fourth bank that uses Ion’s services says clients face a difficult choice. “I think the applications themselves are quite good—very good, I would say, relative to the alternatives,” this person says. “The question then is around trust in the vendor.”

Additional reporting by Josephine Gallagher, Bernard Goyder, Anthony Malakian, Rebecca Natale