New breed of NLP model learns finance better, study finds

A new class of natural language processing (NLP) models trained to catch the drift of sentences, rather than the meaning of single words, may beat the models many investors are currently using to analyze text data.

Academics found in a recent study that a sentence-based NLP model outstripped standard models, including those trained specifically to understand financial terms.

The exercise is a first test, the researchers say, with more work to be done. But the findings suggest quants might wish to consider changing the models they employ.

The results “hint” that the newer algorithms “might be doing a better job”, says one of the co-authors of the study, Petter Kolm, a professor of mathematics at NYU’s Courant Institute of Mathematical Sciences, and a former quant at Goldman Sachs Asset Management.

Most quants build NLP models using Google’s so-called bidirectional encoder representations from transformers (BERT) model as a base.

BERT is an open-source neural network model released in 2019 that comes pre-trained with an understanding of generic language.

Google’s model was trained to guess the deleted words in text excerpts from 11,000 books and the whole of Wikipedia, through more than a million repetitions.

This allows the model to build up a web of so-called word embeddings, mathematical vectors that represent how similar or different words are from one another, taking into account the context a word is used in.

The same technique was used to train FinBERT, a version of the model developed specifically for the financial industry.

SentenceBERT, which academics developed shortly after the release of BERT in 2019, is a modification of the algorithm that looks instead at whole sentences and trains by learning to guess which of a pair of given sample sentences comes first. 

Kolm and his collaborators tested how well models based on BERT, FinBERT, SentenceBERT and other popular models could replicate sentiment scores for news stories from RavenPack Analytics, using only the stories’ headlines. RavenPack is a provider of widely used sentiment signals and event-based data for investing.

The model using SentenceBERT comfortably beat vanilla BERT and a long short-term memory model, another popular type of NLP. Despite being trained using generic data, SentenceBERT also beat a model using FinBERT.

SentenceBERT appears to have a better way of understanding context

Petter Kolm, NYU Courant Institute of Mathematical Sciences

“To our surprise, SentenceBERT performed much better,” Kolm says.

The results suggest context is especially important in how models make sense of financial text, he adds. “And SentenceBERT appears to have a better way of understanding context.”

The researchers calculated the mean squared error for the models’ sentiment scores versus RavenPack’s, with the SentenceBERT-based model achieving a lower error in tests over six-, 12- and 24-month training periods. Compared with vanilla BERT, the model error was roughly halved.

The academics argue the results match the intuition “that for [understanding] financial sentiment, the most important aspect is the quality of the sentence-level embeddings, not the specific structure or vocabulary in the financial domain”. Vanilla BERT’s approach may not be “suitable” for sentiment tasks in finance, the researchers state—“at least not ‘out of the box’”.

A better BERT?

Buy-side quants have used NLP to generate sentiment signals from financial news, earnings calls and company filings, and to classify companies along thematic lines such as those likely to benefit from a transition to low-carbon economies.

Banks have used NLP to keep up with a wave of credit score transitions in the wake of Covid-19.

The findings of the study are empirical and therefore only suggestive, Kolm cautions. “This is not the final word. But it is suggestive of how these models can be improved.”

The experiment used models trained only on headlines, not entire articles. Repeating the exercise with other types of text could garner different results.

“Of the many types of unstructured finance datasets, news may be the closest to a general dataset like the ones SentenceBERT was trained on,” says Nitish Ramkumar, lead data scientist at LSEG Labs, who helps build NLP models for financial services firms.

A model’s knowledge of specialist vocabulary could still prove more important for technical datasets such as earnings call transcripts, company filings, ESG reports, and so on, he points out, though he adds that SentenceBERT could also be fine-tuned using those financial datasets.

There are alternative explanations, also, for why SentenceBERT might do better than other types of models.

Charles-Albert Lehalle, quantitative research lead at the Abu Dhabi Investment Authority, has said NLP models that by necessity compress the meaning of words into vectors of finite dimensions can be prone to errors.

Even the most complex models can represent words in too blunt a manner to avoid associating terms that shouldn’t go together.

Lehalle showed in an experiment that NLP models can ascribe negative sentiment to company names in instances where the company has been subject to litigation, for example.

It’s possible that SentenceBERT beats BERT and FinBERT in NLP tasks for financial text because it captures better and less biased embeddings, Kolm says.