Opportunities, challenges, and strategies for evidence production and use in the COVID era: 3ie’s expert panel weighs in

Evidence is already playing a key role in shaping our responses to COVID-19. But at the same time that the pandemic creates opportunities to broaden evidence use, it also raises numerous challenges for those working to collect, analyse, and synthesize data. About 600 people tuned in Thursday to watch our expert panel on evidence in the time of COVID-19, part of 3ie’s Virtual Evidence Weeks which continue through the remainder of the month.

“Many of these radical policy measures [like lockdowns] to control and manage COVID-19 are made and justified with reference to evidence,” said Birte Snilstveit, director of synthesis and reviews and head of 3ie’s London office.

Nonetheless, there are big gaps in the available evidence, panelists agreed.

“The basic data on death or disease is either missing or hugely incomplete [in many countries],” said Amanda Glassman, executive vice president and senior fellow at the Center for Global Development.

Glassman cited evidence from the UN that in developing countries, less than half of all deaths are even recorded. In countries like Brazil and Peru, the best data on deaths are coming from burial providers, not the health systems.

“We have kind of an emergency on basic data to feed the evidence,” she said.

Much of the discussion focused on strategies to collect data despite the pandemic and lockdown.

One strategy is to use existing sets of phone contacts gathered for earlier surveys, said Iqbal Dhaliwal, global executive director of J-PAL.

“A lot of our researchers have quickly gone back to those banks of phone numbers and called up people to ask quick things [about topics like government responses to COVID],” Dhaliwal said.

Other groups are using innovative research approaches to develop new evidence rapidly, said Norma Altshuler, program officer for global development and population at the Hewlett Foundation.

She cited the example of a Young 1ove, a group in Botswana working with their government to do rapid experiments to improve remote schooling. They are currently doing an experiment on whether teachers should take the time to call households where students are falling behind, or whether they should just text them.

“I think that we have seen some progress in methodological innovations that help with rapid timelines,” Altshuler said.

Dhaliwal also highlighted the “treasure trove” of government administrative data that already exists.

But some of these approaches raise new privacy concerns.

“We already have very clear examples of people using data on mobility to impose punitive lockdowns with all kinds of knock-on effects,” Glassman said.

She’s particularly worried about forthcoming studies using blood data. Such research holds tremendous potential, but “the temptation to misuse [the data] is so high.”

One other potential pitfall is information overload and duplication, Snilstveit said.

“Over the past 3 months over 7,000 papers have been published on the pandemic, one fifth of them in the last week,” she said. “There is an explosion of new COVID initiatives [and] policy briefs.”

Snilstveit said she recently assigned a research assistant to spend 60 minutes finding as many COVID-related guides on a single topic as he could – and he found 69 briefs.

“Over 200 COVID-focused evidence hubs exist … there are at least 10 different trackers focused on tracking policy responses to COVID. So I think that this flow of information at the moment is magnifying the existing challenges of the evidence infrastructure, [including] a lack of coordination between funders, researchers, and users.”

Watch the whole discussion here and join our sessions on 3ie’s new development evidence hub (12 May), importance of cost evidence in impact evaluation (14 May), barriers to integrating Cost-effectiveness Analysis and Cost-Benefit Analysis into impact evaluations (21 May), and how to scale interventions up (28 May).