Rapid Feedback MERL and Lecture Pour Tous Lean Testing Phase 1 Findings Memo

The program aims to introduce Information and Communication Technology (ICT) to enhance and reinforce continuous teacher professional development. Through a partnership with French telecommunications company Orange, LPT will pilot and roll out two sets of ICT tools during the 2018-2019 and 2019-2020 school years. The first set of ICT tools includes a virtual private network (VPN) platform for free calling and SMS between teachers, directors, and inspectors, as well as SMS push messages from LPT. The VPN platform is designed to support communities of practice and peer learning and coaching. The SMS push messages will provide information on timetables and class sequence, use of materials, pedagogical and evaluation techniques, motivation and encouragement, and reminders about upcoming training sessions. The second set of ICT tools includes the delivery of smartphones to directors and tablets to inspectors to facilitate their roles as Coaches, as well as a new Mobile Training EveryWhere (M-TEW) platform to enable Lecture Pour Tous to push ICT content such as text messages, quizzes, voice or video messages, and surveys to support coaching practices. The Rapid Feedback Monitoring, Evaluation, Research, and Learning (RF MERL) consortium is partnering with LPT to use innovative MERL approaches to evaluate these ICT activities as they are piloted in order to improve their design and effectiveness during scale-up. Lean Testing is one approach being used to evaluate the usability and effectiveness of the ICT tools. Phase 1 of Lean Testing was carried out in January and February 2019 during the pilot of the first set of ICT tools. The research questions guiding the Lean Testing were: Are users (teachers and directors) able to use the LPT-provided SIM card? Are they able to navigate their phones and the VPN to send and receive messages and to access SMS push messages? Are they reading the SMS push messages? Are they understanding the content of the SMS push messages? Are they incorporating information received through the SMS messages into their roles? If users are not taking each of the above actions, why are they not, and what could be improved? The RF MERL team used a combination of in-person usability testing, SMS content testing, and semi-structured interviews to answer these research questions. Usability testing aimed to identify potential challenges with implementing the VPN, such as the need for all participants to use an LPT-provided Orange SIM card and to have access to a cell phone and the Orange network. SMS testing was used to determine whether the messages are being internalized in the way that LPT hopes and whether they are likely to lead to desired behavior from the respondent. Semi-structured interviews were used to collect basic demographic information, to understand teachers' and directors' exposure to technology, and to understand if, how, and why or why not teachers and directors have been using the VPN during the pilot phase. The RF MERL team collected data over a two-week period between January 28 and February 8, 2019. The team visited three schools each in the Matam and Kaolack regions, interviewing 6 directors and 11 teachers in total. The sample was designed to ensure coverage across key characteristics of interest, including urban/rural status, national language of instruction, and sex and age of the school director. Key findings from Phase 1 of Lean Testing include: teachers and directors reported difficulty using the VPN and accessing SMS push messages, particularly in rural and remote areas; teachers and directors reported difficulty understanding the content of SMS push messages, particularly those with technical or pedagogical content; teachers and directors reported difficulty incorporating information received through SMS messages into their roles, particularly in areas where they lacked access to technology or had limited experience with ICT. These findings suggest that the ICT tools may require further refinement and testing to ensure their usability and effectiveness in supporting teacher professional development in Senegal.