Broken chains: how DLT code switch compounded ASX fail

November was a bad month for blockchain, even in Australia.

Days after FTX imploded in the US, consultancy Accenture delivered its review of an attempt by ASX to replace the Chess equity settlement system with one based on blockchain provided by Digital Asset. The verdict was damning enough for the project to be “paused” to “revisit the design solution” and DA’s software “derecognized” at a cost of around A$250 million (US$168.3 million).

And Accenture didn’t even get the full story, according to sources.

Technology risk consultant and blockchain skeptic Pat O’Connell says the Chess replacement is a “total failure of strategic technology risk management”: “It is an indictment of the Australian financial community, especially equity brokers who kept silent for fear of looking silly—very few come out of this debacle well.”

The Accenture report found that “siloed execution and reporting between ASX and DA have resulted in misaligned views” of delivery progress and risks, and that the “siloed client/vendor culture” compromised shared business outcomes.

ASX’s announcement prompted a stern rebuke from its regulators, the Australian Securities and Investments Commission and the Reserve Bank of Australia, which called it a “significant setback to the replacement of critical national infrastructure”. They noted the industry has incurred “significant cost to date” on the program.

Even after all this criticism, what surprises the chief technology officer at a major blockchain firm is what Accenture didn’t mention: “Although the technology completely changed during the project, without re-tendering, none of that was covered in the paper.”

Effectively, the project changed digital horses in midstream. The CTO is looking at the lessons learned from the debacle for his own firm’s projects, “so we don’t fall into the same traps”.

“Maybe they selected the right technology for the right vision, but the vision changed, and there was a failure of engagement with the industry,” he says.

A spokesperson for ASX says: “Accenture was not asked to review the history of the project, but to look at the current state and work required to complete.”

In a statement, Digital Asset says it is “in alignment” with the report’s findings. Accenture declined to comment further than the findings of its report.

Hassle with Haskell

ASX initially considered the use of distributed ledger technology (DLT) to replace its Chess settlement system in 2015, and DA was selected for the job in January 2016, at which point ASX also took an equity stake in the fintech. ASX currently has a 5% holding in DA worth A$35 million.

In theory, there should have been a natural shared interest in the success of the project. But the head of execution at an Australian bank suggests that, in the circumstances, owning a stake in DA was a conflict of interest that prevented ASX from making any “hard-faced decision” on the vendor as the project faltered.

DA’s ledger was originally written in computer code Java, which has a developer community of around 14 million, according to research firm SlashData’s Q3 2022 Developer Nation Survey.

After being awarded the ASX contract, DA bought Swiss blockchain firm Elevance Digital Finance, acquiring as part of the deal a functional programming language based on academic code Haskell. Put into production as the Digital Asset Modeling Language, Daml is regarded as elegant in application. Last week, Goldman Sachs launched a new digital assets platform coded with Daml. But Daml’s use is constrained by a small pool of skilled developers. SlashData does not track the use of Daml, but says 500,000 back-end developers use Haskell.

In 2019, ASX brought in virtualization and cloud computing software provider VMware to help DA build out the shared ledger. The beta version of VMware’s own blockchain had been announced less than a year before.

This wasn’t a failure of blockchain—it was a failure of DA and ASX

CTO at a major blockchain firm

Paul Conn, president of global capital markets at stock transfer company Computershare, says the addition of VMware partway through the project to replace DA’s ledger technology was not explained well by ASX at the time, and should have represented “a red flag” that all was not well with the roll-out.

The resulting code for the Chess replacement consisted of around 87% Daml and Haskell-like Daml and 17% in another language, Scala. SlashData says 700,000 back-end developers use Scala, with an additional 900,000 in the machine learning space.

“And that’s what they moved onto,” says the blockchain CTO. “There was a complete change to the underlying technology.”

Scala has similarities to Java, the CTO explains, but is more powerful and more complicated. “So you go from a boring Java code base—that I suspect is what ASX thought they were buying—to a Daml-Haskell-Scala hybrid thing, and the complexity goes through the roof.”

The Accenture report identified support issues for maintenance of the Chess replacement. The CTO believes the composite code would have made those maintenance problems much more severe.

Fixing data issues in the ledger would require custom code by skilled developers, beyond the capabilities of IT teams providing immediate support from ‘follow-the-sun’ locations. Those lower-skilled support teams would therefore need to identify the right expert developers to deal with a ledger failure emergency.

“You’ve got a stack which is Daml, Haskell, Scala and VMware blockchain,” says the CTO. “Where do you start to figure out what’s gone wrong and which specialist to get out of bed?”

The ASX spokesperson says: “Regarding the languages used, we recognize that this is certainly an area of development from a technology perspective, and also that this is a complex system. The project has evolved over time to reflect this.”

Heavy load

Since inception, DA’s ASX project suffered a litany of false starts, and in May this year ditched its latest go-live of April 2023. The software had come under fire on previous occasions. In 2021, for example, clearing brokers complained the move to an exception-only reporting model meant they would no longer receive confirmation messages for trades that settle successfully.

The ASX press release announcing the software derecognition says complexity in the design contributed to challenges in achieving the necessary supportability, scalability and stability for clearing and settlement. Accenture found that settlement under load in the replacement Chess failed with more than 100,000 transactions.

The head of policy at a trade association says: “As with bitcoin, it’s not a particularly fast throughput system.”

Moreover, the system would quit rather than handling certain business errors, and was unable to process transactions concurrently. The head of a DLT firm that works with financial market infrastructures (FMIs) says concurrent processing helps ensure message certainty, which is a must-have feature in capital markets infrastructure.

ASX currently completes 120,000–140,000 net settlements a day, with spikes at 160,000 levels and a prior peak load of 300,000 in March 2020.

“Regulators would normally look for FMIs to have headroom of at least two or three times [the] prior peak,” says Computershare’s Conn.

“That could get you to 900,000 or 1 million net settlements per day, so 100,000 is not really fitting the bill.”

The head of the DLT firm working with FMIs has seen multipliers as high as 10 times the prior peak requested by clients on implementations.

Arjun Jayaram, chief executive of DLT post-trade infrastructure provider Baton Systems, says some of its clients might specify systems that handle 15 million events in a day, with each transaction going through between four and 15 events. That equates to somewhere between 1 million and 3.75 million trades settling each day.

Sunk costs

Although the software derecognition is a heavy cost for ASX, some of its clients say the implications for vendors involved in the project are even worse. The CTO says that vendors would have spent considerable time running proofs of concept, creating design documents, and probably had a version of the software ready.

“Maybe they’ll make that money back because they’ve still got a contract with a bank. But, ultimately, somebody’s going to pay if work is done that gets thrown away, and A$250 million is by far a lower bound on the overall cost to the ecosystem,” he says.

The head of execution at an Australian bank agrees: “The people that have invested the most money in this have been vendors who have had to do the work to get ready. If you talk to GBST or Broadridge, those guys have had 50 to 100 developers getting ready for the project.”

Broadridge and GBST declined to comment.

Maybe they selected the right technology for the right vision, but the vision changed, and there was a failure of engagement with the industry

CTO at a major blockchain firm

The head of execution says his bank accelerated a program of work to get ready for the Chess replacement by buying a new back-office system. According to the contract the bank signed, the vendor is on the hook to facilitate access to whatever ASX decides to replace Chess with.

“The way these contracts work is that the back-office provider is obligated to connect to whatever platform ASX provides,” he says.

ASX says it will use the findings from Accenture’s report to help determine a revised solution. The exchange reckons some parts of what has already been built might be reused.

In 2019, ASX announced that alongside the roll-out of blockchain would be the introduction of the ISO 20022 messaging system, an industry standard for electronic data interchange. The ISO option permits Swift messaging, for example. Swift declined to comment.

The head of policy at the trade association says work done on transitioning Chess messages to the ISO 20022 standard “would be the sort of thing that could stay on, depending on which direction ASX goes”.

Computershare’s Conn—who says his firm had a team of 25 to 30 people working on the project for seven years at “considerable” cost—agrees there may be some value retained in adoption of the ISO global standard. Conn says few in the industry chose to connect to the platform using ‘nodes’ on the ledger, and instead connected through ISO 20022: “But lurking below that is a question as to whether or not all of those messages will continue to be relevant to the next phase of the project.”

Nothing new under the sun?

The blockchain CTO says that, while ISO 20022 messaging is more modern than what Chess uses today, and some of it may be reusable, “it’s a small part of [the overall project], and it’s still a traditional messaging-based system, rather than a shared ledger”.

This goes to the heart of one of the other fundamental questions about the project: was the DLT angle really necessary at all? The Accenture review concluded that, since ASX is the central source of truth and final arbiter of outcomes, it minimizes many of the benefits of a decentralized architecture.

The ASX spokesperson says: “The project did not seek to restructure the market architecture, rather to modernize the infrastructure and improve access to reliable, timely and accurate data, and the opportunity for increased efficiency via multi-party workflow.”

The blockchain firm CTO says that ASX ended up with “a vision for market architecture” that was little different from its existing system: “It was still going to be a fairly centralized ASX-led market with brokers, dealers and custodians effectively just passively taking feeds.”

In addition, he says, cash stock trades would remain settled over two business days—despite earlier claims that settlement times would shrink.

“It sounds like the issue largely came down to the forced application of distributed processing, when a centralized processing system would have been more appropriate,” agrees the head of the DLT firm working with FMIs.

“If you don’t need distributed processing, why use a distributed programming language which introduced other complexities in terms of upgradability, maintainability and hiring developers?”

But the blockchain believers say that, while the ASX fiasco may lead to some short-term collateral damage for the DLT industry, it should not be viewed as a judgement on the technology as a whole. It may even prove beneficial if it prompts other firms to think more carefully about their own projects.

“What people need to assess is whether this is the right tool for them to use in order to do this [task]. DLT is not the right tool for everything,” says Baton’s Jayaram.

The head of execution at the Australian bank is also sympathetic. He says the scope of the ASX project was too ambitious, and that it is still possible for FMIs to adopt blockchain successfully in “niche areas, rather than the core of the system”.

The CTO sums up: “Fundamentally, this wasn’t a failure of blockchain—it was a failure of DA and ASX.”