Testing Tools for Assessing Systemic Change: SenseMaker

The trial focused on the Seed Multiplication Project (SMP), which aimed to build the capacity of a network of small commercial farmers (SCFs) to provide goods and services to smallholder farmers (SHFs). The intervention was expected to facilitate SHFs' transition from older and less efficient to innovative and more profitable farming practices. The research found that the program intervention is strongly associated with change in behaviors of affected SHFs, who are more likely to adapt new farming practices than their counterparts not exposed to the intervention. There is also some diffusion of innovative farming practices among non-client smallholder farmers in the communities where SCFs are present. Continuous and high-frequency interaction between SCFs and SHFs is strongly associated with high rates of change in farming practices. SCFs have a strong influence on the way SHFs work and help reduce costs and increase revenue. Machinery services provided by SCFs make the biggest contribution to farming practices, followed by information-based services. The analysis also identified small clusters of treatment group respondents that are distinct from all other observations. These clusters, often referred to as outliers or positive deviants, may serve as early or weak signals of change when they appear following multiple applications of the tool. Some of the outliers identified in this study relate to the relationships between SCFs and SHFs contributing to diminished workload and both increasing revenues or reduction in cost. The absence of innovative methods of farming and marketing in a specific subset of treatment group respondents was also identified as an outlier. The trial also looked at the utility of SenseMaker itself as a tool for market systems practitioners and evaluators. The findings suggest that SenseMaker has a potential to provide insights into the 'how' and 'why' properties and behaviors in a system change, as well as to identify modulators that affect change (e.g. frequency of interactions). However, there are a number of caveats that need to be taken into account. SenseMaker typically requires supplementation with additional tools in order to inform project interventions. The findings can suggest areas for further investigation, but cannot in themselves indicate how a project should respond. The trial showed that SenseMaker requires time and relies on external support, as well as continuous engagement from the project team in order to generate fruitful evidence. Additionally, SenseMaker is like other tools in that it will not automatically surface systemic changes. Users must have a concept of the types of systemic changes they are interested in understanding during the design phase, so that this can be reflected in the structure of the signification framework. Future applications of SenseMaker would also greatly benefit from a better sampling strategy, project team record keeping of clients, sales, and participants, and a quantitative survey that captures system properties that would help inform SenseMaker instrument design and analytical framework.