Nasdaq, Cboe Fix Feeds Hit by Options Overload

Nasdaq’s BX Options market and the Chicago Board Options Exchange (Cboe) both implemented emergency updates to the software that runs their options market depth datafeeds last month after volatile markets stemming from the Covid-19 outbreak meant the exchanges literally ran out of sequence numbers to assign to orders in their systems.

On 10 days during February and March—February 25, 27, and 28, and March 2, 3, 4, 6, 10, 12, and 13—Nasdaq warned market participants that the BX Options Depth Feed would exceed its sequence number threshold and would not function normally for the remainder of those days.

The feed’s sequence number limit was 4.3 billion messages per day. On those occasions, when the depth feed ran out of numbers to represent options order activity in Nasdaq BX’s system, the feed would begin a fresh cycle starting at message number 1 again.

“The way we designed the feed when we went live with BX Options in 2013, there was a limit that we never anticipated hitting,” says Greg Ferrari, head of US options at Nasdaq. “When the max sequence level was reached, it meant that market participants didn’t have full visibility into the BX Options order book, but they still could see auction data and top of book data from their other order book feeds and protocols available in the market. Cycling back to a previously delivered depth of market sequence number in the same trading day wasn’t an option, so we had to shut it off on those days once the level was reached.”

In the absence of the market depth feed, firms could continue trading because BX’s TOP (top-of-book) feed was unaffected. Most firms also consume the TOP feed along with the depth feed. There was one firm that only took the depth feed, without the top-of-book feed, and that firm was able to make coding changes to mitigate the issue, officials say.

“Depth allows for firms to have full coverage of existing liquidity on our order book, which adds value … rather than being something that you trade on,” Ferrari says. “If you have a customer order sitting one tick away from the best bid or offer, that’s insightful data, but you can’t actually trade with that resting order without clearing local and away better-priced interest because of Reg NMS.”

To provide a permanent fix, Nasdaq needed to perform a “specification change”—a change that requires clients to make changes—which it orchestrated over a period of three weeks from origination to completion, whereas a similar change would usually require the exchange to give three months’ notice to allow clients to budget resources in advance of the change.

Nasdaq changed its message number formats from four-byte to eight-byte in a “hot cut” migration that took place in two phases. On Monday, March 16, Nasdaq began publishing the updated version of the depth feed over its New York Metro A and B feeds, and maintained the legacy version of the feed from its Chicago disaster recovery site as a backup through Friday, March 20, before cutting over to the new version the following Monday, March 23.

Nasdaq made test files available for clients to test the changes beforehand. “The feedback from clients was that it wasn’t a single day’s work, but it didn’t take a whole week … so it was not that disruptive [for clients],” Ferrari says. “The spec change allowed us to use a different character limit level that allowed us to go beyond 4.3 billion messages per day. This is a permanent fix. … Now we have a lot more headroom in terms of how the machine outputs numbers.”

During the affected period, Nasdaq was in “constant” contact with the Securities and Exchange Commission (SEC), to keep the regulator appraised of the issue and Nasdaq’s plans. “We continuously raise our hand with the SEC. We’re in constant dialogue with them about any and all issues,” Ferrari says. “So we told them that we had it under control, that we had a plan to fix it, and that we wanted to be considerate in terms of how we approach it, and that firms have a workaround in the meantime so they could continue to provide liquidity to this marketplace.”

He adds that Nasdaq used the experience as an opportunity to evaluate its maximum sequence numbers across all its other marketplaces, and—aside from its PHLX Options (the former Philadelphia Stock Exchange) market, where it plans to deploy a similar fix—is confident that all have sufficient headroom.

Cboe experienced similar problems with the current four-byte decimal sequence numbers supported by its market depth feeds, which had a limit of just under 4.3 billion messages per day. The exchange first reported issues with the engine running its SPXW options contract (options on the S&P 500 index that have weekly expiry dates) over its Multicast Pitch and TOP datafeeds on February 28, warning market participants as sequence numbers approached maximum usage, and was able to deploy a patch on March 2 that would allow systems to continue operating in the event that the maximum number of messages was exceeded, and promised to provide future dates for testing and deployment of a permanent fix.

On March 2, the feed exceeded 70% of the maximum allowable messages, and on March 3, the feed surpassed 95% of the maximum message numbers. In each case, the exchange notified market participants that even if it was forced to take the feed offline, other systems and feeds would continue operating normally. 

Cboe did not respond to questions prior to publication. We will update the story if and when it does.