How do we know stats can be trusted?

In April, as the coronavirus pandemic was gathering force around the world, reporters asked Brendan Murphy, then Australia’s Chief Medical Officer, about international case numbers. In his answer, he made an interesting comment:

“The only numbers I have total faith in are the Australian numbers.”

“Faith” might seem an odd word to use about about official statistics. But in our research, which involves talking to statisticians, public servants and journalists who produce and communicate the statistics that govern our lives, people say overwhelmingly that faith and trust are essential parts of what makes statistics useful.

Despite the objective and impartial appearance of statistics, it is a web of people and human processes that makes them trustworthy.

When trust falters

The COVID-19 pandemic has shown us the importance of numbers we can rely on. In many cases, early problems in data management have been indicators of bigger problems to come.

In the United Kingdom, a lack of local case data was hindering public efforts as early as June. By October, it emerged that some 16,000 cases had been lost from national reporting due to sloppy data management.

In Australia, the quality of COVID-19 responses in different states has broadly reflected existing perceptions of the quality of each state’s published data.

New South Wales was celebrated early on by data scientists for its responsiveness and transparency in data management, long before Prime Minister Scott Morrison’s September comment that the state’s contact tracing was the “gold standard”.

Meanwhile, inconsistencies in Victoria’s data management — such as irregular definitions of “local” and “community” transmission, and confusion over the definition of “routine testing” — were rankling data scientists back in June, before the state’s mystery cases blew up into a second wave.

Governing by numbers

Public statistics play a huge role in our lives. Political staffers we have interviewed in our research explained how the “mere presence of any kind of numbers” seems to make people more trusting of any claim.

Enthusiasm for decisions made “on data” often relies on the image of statistics having what philosophers call “mechanical objectivity”. By this logic, statistical results are essentially produced by machines or mechanical processes, so decisions based on data can be routine, apolitical, unbiased and incontestable.

But talking to the people behind our national numbers reveals how official statistics are in fact the product of expertise, immense resources, personal judgement calls, and trust.

Analysts we spoke to described official statistics as “icebergs” concealing vast tangles of sources and stories. Statisticians often used words like “instinct” and “vibe” in explaining how they check data sources and rely on long-term relationships to gather figures. One public servant summed up the situation neatly:

“A lot of personal relationships are stringing together government statistical work.”

The process of transforming real life into tables of data takes expert human intervention at every stage. These interventions can result in vastly different answers to the same initial questions. As the answers often form the bedrock of public decision-making, it becomes crucial we can trust the people who make the decisions behind the scenes.

“Total faith” in Australian numbers

When the Statistical Society of London was founded in 1834, discussion of interpretation or “opinion” was banned at its meetings. Modern statisticians have a much greater understanding of the importance of how people feel about data: the theme of this year’s World Statistics Day was “Connecting the world with data we can trust”.

But trusting “data” buries the work of the people who create statistics. In practice, they are the ones we are trusting.

In Australia, at least, we trust them a lot. Murphy’s “total faith” in “Australian numbers” is reflected in a 2020 survey conducted by the Australian Bureau of Statistics, which found 85% of the general community trust ABS statistics. The figure is even higher among “informed users”: a near-unanimous 99%.

Our communal faith in the numbers produced by the ABS reflects a web of trust and relationships built over decades.

Why trust matters

For many policy issues, such as unemployment, ABS statistics are taken as our absolute measure of reality. In these cases, the fact that 99% of informed users trust the numbers is what lets government and society keep functioning.

ABS data forms a shared basis for discussion. One analyst told us it’s not just trust in the bureau’s expertise that creates trust in the data — it’s also “because everybody else is using it as well”.

Those who work with public statistics agree these networks of trust — both between institutions like the ABS and the Australian public, and between individual academics, public servants, statisticians and journalists — rely on day-to-day honesty and responsiveness.

Transparency and consistency are essential. Statisticians and journalists alike emphasised that even data formatting plays a role in building trust. One journalist we interviewed talked about a health agency releasing coronavirus statistics as PDF files emailed to a private list early in the pandemic, describing it as “a ridiculous situation” that created extra work for the journalist to make the numbers clear and accountable in public. (State governments have since improved on this, generally making daily case numbers publicly available online.)

Long-term trust

Our response to future crises, whether the next pandemic or the impacts of climate change, will rely on the strong communal web of trust in our statistics.

Statistical failures, like the UK losing 16,000 cases or the United States massively undercounting coronavirus deaths, damage public trust in government. When institutions can prove their statistics are trustworthy, the public are more willing to follow directives based on those statistics. How do we know we can trust statistics? We talked to the humans behind the numbers to find out

Samantha Vilkins, PhD candidate, Australian National University; Sujatha Raman, Associate Professor, Australian National University, and Will J Grant, Senior Lecturer, Australian National Centre for the Public Awareness of Science, Australian National University

This article is republished from The Conversation under a Creative Commons license. Read the original article.