Key Strategies for Measuring Reach, Engagement, and Outcomes of Distance Learning Initiatives

The document identifies comprehensive measurement approaches to ensure quality, equity, and inclusion in distance learning. Distance learning is defined as teaching and learning where educators and learners are in different physical spaces. It can be designed for learners of all ages and levels, from preschool to adults in nonformal learning programs or higher education. Distance learning can also be designed for educators, caregivers, and other family members. The brief highlights the importance of measuring reach, engagement, and outcomes to ensure the effectiveness of distance learning initiatives. A three-pronged approach is recommended for measuring distance learning: integrated monitoring, evaluation, and learning (MEL) approaches, multi-modal technology interfaces, and mixed methods measurements. This approach helps ensure equity, inclusion, and quality of data. Integrated remote and in-person data collection enables more frequent, responsive, and systematic data collection in emergency and non-emergency contexts. Remote data collection provides timely data on reach and engagement, while in-person data collection is preferable for measuring outcomes, including attitudes, beliefs, and behaviors. Mixed methods data collection combines quantitative and qualitative data collection methods, allowing for deeper analyses and greater opportunity to measure intended and unintended reach, engagement, and outcomes. Quantitative data collection methods include surveys, tests and assessments, teaching and learning analytics, and observations, while qualitative data collection methods include qualitative observations, focus group discussions, interviews, and participatory and arts-based research. Multi-modal technology interfaces are used to measure distance learning through multiple technology interfaces, helping to reach a wider group of participants, including those with limited access to technology and connectivity. Interfaces are selected based on technology device access and accessibility needs, connectivity, and demographics of the users. Technology interfaces include phone or video calls, interactive voice response, text messages, social media groups, paper, images, video and audio recordings, learning management systems, and educational apps, programs, or games. The Briefing Note provides a hypothetical case study of a distance learning initiative, where digital analytics programmed into mobile phone applications are used to measure reach and engagement, as well as some knowledge outcomes. In-person one-on-one assessments with learners are used to measure literacy and numeracy outcomes, as well as socioemotional outcomes. For learners, caregivers, and educators who do not have phones, surveys and interviews are conducted in-person when feasible. The document emphasizes the importance of using a combination of methods to ensure inclusion of all learners and to identify intended and unintended outcomes. For households with phones, periodic mobile phone surveys are conducted with learners, caregivers, and educators to measure reach and engagement in the distance learning programming. Caregivers and learners share student work and ask educators questions in virtual groups, such as WhatsApp, to measure socioemotional learning. Educators conduct phone interviews with caregivers and learners to measure socioemotional learning.