Historical significance is hard to predict

Can events be accurately described as historic at the time they are happening?

Over the years, philosophers have tended to argue that the answer is “no”; now modern research using digitised data from the US State Department suggests they are right.

Writing in the journal Nature Human Behaviour, a team from Microsoft Research and from Columbia University in the US say a combination of competition for scarce attention and other factors inherent to human systems poses difficulties for predicting political events, success in cultural markets, the scientific impact of publications and the diffusion of information in social networks.  

Two centuries ago, German philosopher GWH Hegel argued in his Philosophy of Right that philosophical understanding of an era is possible only as its ending.

However, it was similar sentiments in the more recent work of US philosopher and art critic Arthur Danto – Analytical Philosophy of History, published in 1965 – that inspired the latest study.

“It was one of the densest books I’ve ever read but its thesis was remarkable: that the meaning of historical events cannot be known at the time they are happening,” says lead researcher Duncan Watts. 

In his book, Danto argues that even if we assume the existence of some super-being who knows everything that has ever happened and everything that everyone is doing and saying and thinking right now that being still couldn’t tell you the meaning of an event that historians will one day ascribe to it. 

The storming of the Bastille, for example, was just one event among many in the heady days in the summer of 1789. Its full significance, Watts says, only became apparent months or even years later as the French Revolution transformed first France, then democracy, and ultimately world history. 

“I recall being stunned by this realisation because, if true, it upended some of our deepest assumptions about human rationality,” he says. 

The power of rationality, he adds, is that it can both make sense of our past actions and predict the future consequences of the decisions we are making now. 

The opportunity to see whether Danto, and others, were right arose when Watts met Matthew Connelly, a Columbia historian with access to an unusual dataset – nearly two million bits of State Department data archived between 1973 and 1979. 

With Microsoft Research colleagues in the US and India he analysed the data using a variety of machine learning models.

They used a corpus in which each document corresponded to a single event, allowing them to evaluate documents rather than events directly.  Finally, each document was assigned two “importance scores”: one by a contemporaneous observer and one, decades later, by a future historian. 

On balance, they say, their results suggest Danto was “substantively correct”. Historical significance is extremely hard to predict as the number of candidate events grows large. 

“As the number of events being evaluated grows, successful predictions will be increasingly outnumbered by events that seem insignificant at the time, but which come to be viewed as important by future historians in part because of events that have not yet taken place,” they write.

A further complication, they note, is that historical significance, even when it can be meaningfully assigned, is specific to observers whose evaluation may depend on their own idiosyncratic interests and priorities. 

“Although we speak of history as a single entity, in reality there may be many histories, within each of which the same set of events may be recalled and evaluated differently,” they write

“Despite these difficulties, we close by noting that research using algorithmic predictions may nonetheless prove useful to historians in practice.”