The global lockdown measures brought on by the COVID-19 pandemic, resulting in most employees working from home, demand that we reconsider the conventional workplace model and what this model may look like in the future. These considerations may be particularly important for monitoring and evaluation (M&E) practitioners given the resulting shift in the intervention environment. The evaluation of interventions offered by NGOs, governments, international organisations, and other institutions will be critical in prioritising and reallocating funds both during and after the pandemic. Additionally, programmes that are directly related to emergency healthcare are critically important at this time and evaluations of these interventions will be vital in helping improve both current and future outputs and outcomes, and eventual impact.
Lessons for M&E from the 2014 Ebola outbreak
The Ebola Virus disease outbreak has provided much insight for public health officials in developing COVID-19 response plans. Looking at how international organisations and other institutions implemented monitoring and evaluation plans during that crisis, we can learn some key lessons on how best to implement similar research plans in facing COVID-19, albeit a more widespread and sustained disease.
When the Ebola outbreak occurred in 2014, the 10 West African countries that would be affected by the disease were caught unprepared, resulting in over 30,000 Ebola cases, and approximately 11,000 deaths[1]. The outbreak also revealed that the underlying technical, institutional, and human systems and processes required to gather, analyse, and use data were not robust enough to support a timely response.
From an evaluation perspective, the largest challenge was due to poor infrastructure and data management.[2] Specifically:
1. Gaps in reliable electricity and digital connectivity often hindered real-time information sharing.
2. Paper-based data collection was often slow and incomplete.
3. There was a three-week lag between the collection of paper-based case data and their eventual reporting, greatly hindering evidence-based decision making.
4. Key datasets were often published in non-duplicable formats and in difficult-to find locations, limiting primary data usage and secondary analysis for emergency, scientific, or research purposes.
5. Data elements were not globally or locally standardized, making data non interoperable and preventing data comparisons, greatly weakening the response and understanding of the outbreak.
6. Many digital data systems deployed were not sustainable beyond the response due to solutions not being generated locally. This was due, for the most part, to limited activity timelines and limited investment in building local country capacity to maintain systems.
Recommendations
In attempting to advance monitoring and evaluation practices during the COVID-19 pandemic some best practices are suggested based on the work currently being done by organisations globally, as well as the lessons learnt from the Ebola outbreak:
i) Evaluation Innovation: Many interventions and evaluations have been postponed or cancelled because of the COVID-19 pandemic and resulting global lockdown measures. This provides evaluators with an opportunity to innovate around evaluation plans, designs, methods, and users. Programme designers and implementers also have an opportunity to re-imagine existing impactful interventions that can be expanded to remote implementation, in order to extend programme reach. Such innovation could influence how programmes are designed in the future and provide an opportunity to amplify programme reach. A recent example of this is the UNDP’s adoption of real-time evaluations, and the creation of knowledge products that will expedite the synthesis of lessons that can be applied to interventions to recover from the COVID-19 pandemic.[3]
ii) Virtual Engagements: Of the interventions that will continue throughout the various lockdown periods globally, many will have to occur remotely. Data for these interventions can be collected using virtual interviews, USSD questionnaires, online surveys, or via collaboration platforms (such as Slack or Yammer) and satellite imagery[4]. This necessary shift from paper-based to digital data collection can increase the accuracy and reliability of the data collected, especially if data collection tools are programmed to validate the data. Further, digitised data provides the opportunity for real-time information sharing and data standardisation, that would enable its use in research. Stakeholders and evaluators will have to effectively communicate through remote arrangements and adjust deliverable timelines accordingly.
iii) Digital acceleration: Perhaps the greatest potential opportunity provided by the COVID-19 crisis centres around the chance for organisations to advance their digital practices. In addition to forcing organisations to embrace digital modes of data collection, organisations should improve their digital record keeping. Evaluators can assist stakeholders in developing more robust and sustainable monitoring practices during this time, which will in turn enable more effective evaluations in the future.
iv) Sharing evaluation practices among networks: With the world engaging in various levels of physical distancing and the resulting absence of conferences and symposia, novel means of engagement among evaluators should be sought. Sharing evaluations will help to diffuse information on best practices more broadly. This will assist in keeping evaluators abreast of evolving evaluation approaches, methods, technologies, and tools that are required to continue work during crises of this nature.
References
[1] World Health Organisation, 2020. https://www.who.int/bulletin/volumes/92/12/14-149278/en/
[2] USAID, 2018. https://www.usaid.gov/digital-development/covid19-remote-monitoring-guide
[3] +[4] http://web.undp.org/evaluation/mediacentre/infographics/evaluation_covid19.shtml
[5] https://sidw.org/sites/default/files/USAID_DigitalPreparednessConsiderations_v1.pdf
[6] https://usaidlearninglab.org/sites/default/files/resource/files/dn_-_adaptive_management.pdf
[7] https://www.poverty-action.org/blog/covid-19-how-ipa-responding-adapting-and-preparing
[8] http://www.fao.org/evaluation/highlights/highlights-detail/en/c/1268896/
[9] https://blogs.worldbank.org/impactevaluations/impact-evaluations-time-covid-19-part-1