The nine worst Covid-19 biases

We all suffer from cognitive biases that cloud our judgment and lead us to the wrong conclusions. But now that we are in the middle of a pandemic, and restrictions are being put in place that have a profound impact on people’s lives, it is more important than ever that we look to the evidence and challenge these biases before they lead to serious mistakes in our response to the disease.

Unfortunately, this has not been happening. Here are nine big mistakes that have shaped our response to Covid-19:

1. Herd thinking

From the offset, the government’s thinking about the pandemic was based on influenza modelling. The clue is in the title of the most influential modelling group which reports directly to the government’s scientists: the Scientific Pandemic Influenza Group on Modelling (SPI-M). The focus on influenza was cemented in 2016, when the Department of Health ran a cross-government exercise to test the UK’s response to an influenza pandemic, called ‘Exercise Cygnus’. Cygnus showed that an influenza pandemic would cause the health system to collapse, and accordingly the UK’s ‘National Risk Register’ focused solely on the spread and impact of a new flu strain. This reflected the narrative of the last two decades, when influenza was the only pathogen in town. This distorted minds in government into believing that pandemic preparedness was all about antivirals and vaccines (the more the better) which exist only for influenza. Other methods of disease prevention such as barriers (masks, gloves, visors and goggles), personal hygiene and separate infection facilities to house the sick were largely ignored, with consequences that are now clear. When newly-identified respiratory agents such as SARS and MERS emerged, countries in East Asia which were hit hardest by these new diseases prepared accordingly. The UK did not.

2. Confirmation bias

Theresa May’s quip in the House of Commons last week that data should be used to support policy and not the other way round, sums up the way the government is currently handling this pandemic. Whenever the government receives new information, instead of viewing it objectively, it uses the information to support its pre-existing views.

Ross Clark, for instance, argues that the Welsh experience doesn’t support the case for a four week lockdown in England where cases are coming down in many areas and levelling in others. Will the devolved nations take notice of their differing policy initiatives and the impact on case numbers as they plough their own furrow?

Francis Bacon characterised this attitude, writing: ‘once a man’s understanding has settled on something (either because it is an accepted belief or because it pleases him), it draws everything else also to support and agree with it.’

Confirmation bias is not just limited to the government. Both the Lancet and the New England Journal of Medicine have been forced to retract articles, after they were too hasty to accept studies where the data could not be verified.

When you are confounded by confirmation bias, it becomes very difficult to change tack – even in the face of clear opposing evidence.

3. One-way street bias

No matter how partial and unstable the data, increases in coronavirus cases have led to restrictions being imposed on our daily lives. In each case, the restrictions have been justified by scenarios from unverifiable assumptions. If cases are going up, the answer has nearly always been to tighten restrictions.

When asked why this was the case by the Select Committee of Technology and Science last week, Sir Patrick Vallance told MPs that Sage do not consider the economic costs of the policy they are advocating. ‘This sits in Her Majesty's Treasury,' he told the committee. This has led to scary and unbalanced predictions.

The growing errors have led to the government overestimating the number of people who will die from the disease, miscategorising deaths from Covid, giving a misleading impression of Covid hospital admissions and recently using out-of-date models to scare the Prime Minister into the latest lockdown. The data published by the government machinery only ever seems to involve the worst-case scenario.

4. Spin bias

The presentation of future coronavirus scenarios based on predictive models from five different academic institutions has led some people to believe that models are infallible – and so distorted their perceptions of risk. The usual scientific processes of publishing peer-reviewed work have disappeared during this pandemic. Instead, academic institutions produce dramatic results which are presented through press releases. This presentation of conjecture as fact alters the interpretation of results and leads to misleading conclusions.

5. Popularity bias

This arises when there is a particular public interest in a specific disease or treatment, such as coronavirus or lockdown – and it results in a distorted evidence-base. In 2019, there were only 883 coronavirus articles indexed on the medical publishing database, PubMed. So far in 2020 the total has risen to over 50,000. In contrast, there were roughly 6,000 publications on influenza in 2019 and this number has not risen in 2020. The increased awareness has made Covid ‘fashionable’ to some scientists, to the detriment of quality of research.

The twin obsessions with influenza and forecasting have led to sizable gaps in the scientific evidence – both in the determinants of transmission and of the effectiveness of containment measures.

No European government has instituted a serious and well-resourced investigation into the ecology of all known respiratory viruses or the determinants of explosive epidemics like the one in Lombardy starting in late February. Nor have they launched investigations into subsequent episodes of high transmission. Clinical trials of drugs and vaccines have been richly funded during this pandemic but there is no available good quality evidence on whether, for example, masks prevent transmission of Covid-19 in the community and (if so) which types. We could only find three registered trials on the use of masks in the community: one in Denmark, one in Guinea Bissau and one in India – none have yet reported outcomes.

When a topic is fashionable, investigators and experts may be less critical, editors of biomedical journals may not be able to resist the temptation to publish results, and journalists rush out headlines no matter how preliminary or shaky. We have lost count of the number of experts and different opinions and views expressed on all aspects of the pandemic. Divisions are not helping, but there are few who look at the data and try to make sense of what is going on. In this pandemic, social media has allowed overnight experts to flourish under the warm glow of attention. Once the heat has died down most interventions do not work quite as well as first thought.

6. Spectrum bias

When a diagnostic test is studied in one setting – for example, in hospitals – and not in the intended population for which it is used – the community – problems with interpreting the results occur. Mass testing has taken off and the limitations of widely-used tests such as PCR have been ignored. PCR tests on their own cannot distinguish whether you are actively infected and infectious or whether you are recovering from the disease and simply have dead particles of Covid in your system. Up to a year ago a handful of laboratories were doing PCR – a highly delicate and specialised technique, vulnerable to contamination. The enormous expansion of testing in the community has meant procedures and training have been rushed through. Warnings by bodies such as the College of American Pathologists of the problems of using different uncalibrated PCR kits have been ignored. The detection of ‘cases’ in the community tell us little (apart from driving lockdowns). How many are serious; how many are contagious? No one knows.

7. Swamping bias

Swamping bias is the sheer volume of unchecked, under-analysed and misreported information which goes unchallenged. Distortions arise from the use of information which is most readily available. For example, the 4,000 deaths a day worst-case scenario touted by the Chief Scientific Advisor and the Chief Medical Officer was the only information available to the public last week, and it led to the imposition of a nationwide lockdown. Since then, the statistics watchdog has rebuked the government for using this projection.

This swamp of information makes it impractical to properly examine the models which the current restrictions are based on. We could only find this one page document, and this descriptive summary which attempts to explain the government’s recent decision to lock down. Consequently, the methods governments used to justify lockdown are unverified and errors are seldom corrected. No one ever owns up to mistakes, and the mire of the pandemic hinders lessons being learned.

8. Reporting bias

The selective disclosure or withholding of information distorts reality. Essential hospital admission data, for instance, in the NHS is confidential. Like layers of an onion, there are several current realities to this pandemic, around the real viral circulation and its impact, its popular perception based on government broadcasts and its reporting by the media and their experts. ‘Major planning is kept secret even from senior members of the government,’ and vital planning documents often need to be leaked to make it into the public domain. This lack of information is deeply troubling: we know almost nothing of the first reality, little of the second and are overwhelmed by the third layer. Perhaps, like onions, we should hide underground – as the government wants us to – until this is all over.

9. Intuitive bias

The fact that youngsters have hardly been affected seems to have escaped decision-makers. There may be a good explanation for why the virus has passed young people by: they are usually exposed to a variety of other viral agents including common cold coronaviridae and they may have strong cross-reactive immunity. The psychologist and economist Daniel Kahneman in his book, Thinking, Fast and Slow, refers to ‘system one reasoning’ that operates automatically and intuitively, with little or no effort; and for those decisions that do not make sense ‘system two reasoning’ that is more thoughtful, slow and analytical. System two reasoning would set off a search for the evidence to understand the lack of impact in young children as opposed to intuitively locking them up because it feels right.

Biases distort the analysis and interpretation of evidence and our reasoning. The biases we have outlined impede our ability to discuss a reasonable way out of this pandemic. We need an adult discussion about whether we can learn to live with the risk of being run over by a car or falling from a skyscraper or we want to abolish all vehicles and demolish all buildings which are more than one story high.

This post is dedicated to the late David Sackett, the father of EBM, who in 1979 proposed the continued development of an annotated catalogue of bias as a priority.