Final impact evaluation of the PEP pilot radio broadcasts

PEP, formerly called the Improved Efficiency of Learning (IEL) Project, was designed to overcome a shortage of adequately trained primary school teachers, textbooks, and other educational materials. The program includes 510 instructional modules (or lessons). Each module is comprised of reading booklets, practice booklets, and review booklet tests to determine how well students have mastered the material of each module, block and semester. A program to train teachers in the methodology of the system comprises an additional component. In 1989, 60 in-service PEP pilot radio programs were produced to follow-up on the initial training by reinforcing elements of the programmed instruction modules for selected groups of PEP teachers (second-grade reading and fifth-grade science teachers) and principals. The broadcasts began on 6/5/89 and ran during that semester and the following one. In 1989, the Academy for Educational Development (AED) contracted with Applied Communication Technology (ACT) to conduct an evaluation of the pilot radio programs. The purpose of the evaluation was to determine the programs' impact on the administrative performance of school principals and the teaching performance of second and fifth grade PEP teachers. To measure changes in job performance, self-reported information was collected from the target groups using a pre-test/post-test design. Because it was not known prior to the collection of the data whether it would be possible to arrive at quasi-equivalent intervention and control groups, as many of the 360 PEP schools as possible were included in the sample. The pilot radio programs were evaluated by testing teachers and principals both before and after the programs were broadcast, to see if there was any detectable difference in what they did as PEP teachers after the broadcasts. It was found that the radio programs alone (after removing the impact of in-service supervision visits) did indeed help to improve about one-third of the skills that were measured. The specific instances in which this occurred were helping principals to handle teacher shortages and to prepare for and monitor faculty meetings; helping second-grade teachers to read poetry and identify vowel sounds and word sounds; and helping fifth-grade teachers to prepare for class by taking module booklets home and to administer tests. The combination of supervision and radio improved teachers' and principals' performance on almost every item except those on which the initial score was so high to begin with that there was little room for improvement. Skills which improved as the result of a combination of radio and supervisory visits were, for principals, keeping attendance records, taking responsibility for module and test booklets, and holding faculty meetings more frequently; for second grade teachers, going to the principal or Ministry for more booklets, taking modules home to prepare for class, and providing remedial help to students; for fifth-grade teachers, providing remedial help to students, using PEP methodology to promote and monitor student progress, and forming peer groups. In several of these instances, groups which lagged behind on the pre-test were brought up to the level of the other groups on the post-test. In the case of helping second-grade teachers pronounce vowel sounds, results indicate that supervisory visits made an impact, but that the radio programs did not. The results of the evaluation indicate that radio and supervision play an important role in improving the performance of those teachers and principals whose skills are not up to the standard of the group as a whole. Several cases were observed in which the score of a group which was notably lower on the pre-test rose to the level of the other groups on the post-test: principals keeping attendance and taking responsibility for booklets, and second-grade teachers turning to principals and the Ministry for additional booklets. Almost all of the teachers and principals (99% or more) reported that they liked the radio programs, that the programs were helpful, and that there should be more programs next semester. On the basis of these positive (though possibly inflated) assessments, and the improvements in performance that have been recorded in this report, the PEP Program should look favorably on the continuance of radio broadcasting to PEP schools. (Author abstract)