A yet-to-be-peer-reviewed paper by three economists is kicking up a storm by claiming the COVID-19 lockdowns scarcely reduced deaths, yet came at a huge cost to society. 

While the paper has been jumped on by those looking to push an anti-lockdown narrative, many independent public health experts (note: scientists, not economists) have highlighted holes in the study and claimed its findings should be taken with a hefty handful of salt.

Commenting on different parts of the paper, scientists have described it as everything from “strange” and "problematic” to “fundamentally flawed.” 

The economists from Johns Hopkins Institute for Applied Economics, Global Health, and the Study of Business Enterprise reached their findings by carrying out a meta-analysis of 34 previously published studies.

The controversial report – which has not yet been peer-reviewed or officially published – concluded that the initial stay-at-home lockdowns in the US and Europe reduced COVID-related deaths by only 0.2 percent. They also argued that the closure of non-essentially business was “ineffective, only reducing COVID-19 mortality by 2.9 percent on average.” On top of this, the paper contends the lockdown measures came at “enormous economic and social costs.”

However, many scientists have questioned the choice of studies used in this meta-analysis, as well as how much "weight" the researchers gave to some of the studies.

Above all, the paper appears to have ignored research into disease transmission, focusing instead on other statistical analyses of mortality used by economics.

If you’re studying a virus and how it kills people, disease transmission is a pretty crucial thing to consider. For instance, it’s important to realize that stay-at-home orders will have a delayed effect; hospitalizations and deaths won’t instantly fall, but they may drop over the course of two or so weeks. This study, experts claim, fails to recognize this important issue

“It’s as if we wanted to know whether smoking causes cancer and so we asked a bunch of new smokers: did you have cancer the day before you started smoking? And what about the day after? If we did this, obviously we’d incorrectly conclude smoking is unrelated to cancer, but we’d be ignoring basic science,” commented Dr Seth Flaxman, Associate Professor in the Department of Computer Science at the University of Oxford. 

“The science of diseases and their causes is complex, and it has a lot of surprises for us, but there are appropriate methods to study it, and inappropriate methods. This study intentionally excludes all studies rooted in epidemiology – the science of disease,” added Dr Flaxman.

There are also some issues with the definition of a "lockdown” in the paper, which the researchers define as any “mandatory non-pharmaceutical interventions.”

Professor Samir Bhatt, a Professor of Statistics and Public Health at Imperial College London, notes: “This would make a mask wearing policy a lockdown. For a meta-analysis using a definition that is at odds with the dictionary definition […] is strange.”

It also needs to be considered that no two countries "locked down" in the same way with different governments going for significantly different approaches.

Many scientists still agree that social restrictions are effective at helping to slow down the transmission of COVID-19, albeit perhaps not as effective as initially thought. However, the impacts of lockdown are extremely tricky to gauge and fraught with complexity. In essence, it comes down to a question of costs versus benefits. Most agree that confining people to their homes may have reduced their chances of spreading COVID-19, but at what cost?