An HC3 Research Primer: Propensity Score Matching

It allows researchers to determine whether a program was responsible for changes in knowledge, attitudes, and behaviors. Impact evaluation of SBCC programs requires comparison between what happened as a result of a program and what would have happened in its absence. Randomized control trial designs accomplish this by randomly assigning some people to receive a treatment and others to not receive it, and comparing the results between the two groups. However, in large-scale SBCC programs, such as entertainment-education programs that use mass media to reach a national audience, it can be impossible or undesirable to prevent some people from receiving the messages simply to create a counterfactual condition for evaluation. People who hear or see program messages may be different from those who are not exposed, due to factors such as affluence, education, motivation, or predisposition to participate in the program and respond to it. PSM provides a way to take these differences into consideration and control for them when calculating program impact. The PSM technique requires survey data, which should contain measures of things that make a person more likely to be exposed to the program, such as age, gender, language preference, household income, place of residence, prior behavior, access to media, and other demographic and lifestyle factors. Multiple regression analysis is used to identify which of these characteristics is most strongly related to program exposure. PSM then matches people in the survey sample who have the same characteristics that make them more or less likely to be exposed to the intervention, and compares the extent of behavior change among similar people who were exposed (the treatment group) and those not exposed (the matched comparison group). PSM gives researchers confidence that the only difference between the matched persons is the one they want to examine: exposure to a specific SBCC intervention. This allows researchers to evaluate behavior change while controlling for the variables that predispose some people to be exposed and to change. Without assigning some people to receive the program and denying it to others, researchers can be certain that the predisposing variables are not the reason that an individual responded positively to an SBCC program – rather, it was the program itself that had an effect on the individual's behavior. The Scrutinize campaign in South Africa, developed in 2009, is an example of the use of PSM to measure impact. The campaign used propensity score matching to evaluate its effectiveness in increasing awareness of risk behaviors related to HIV infection. The analysis found that being able to correctly recall the risks associated with multiple sexual partners from the campaign resulted in a 3.2 percentage point decrease in the likelihood of having multiple sexual partners. PSM enabled the evaluators to estimate that 3.2% of the population, or over 111,000 people, avoided multiple sexual partners as a direct result of exposure to the Scrutinize campaign. Propensity score matching produces strong evidence of a causal relationship between an SBCC intervention and behavior change in large population-based observational studies. It is a statistically sound technique that can be used when randomized control trials are either not ethical or possible. However, PSM analysis can only account for the variables that have been observed and measured by a particular survey. If there are other variables affecting the relationship between the intervention and behavior change not measured or not identified in the survey data, then PSM will not be able to account for them.