Evaluating Educational Interventions That Induce Service Receipt: A Case Study Application of City Connects (2016)

Educational interventions are complex: Often they combine a diagnostic component (identifying student need) with a service component (ensuring appropriate educational resources are provided). This complexity raises challenges for program evaluation. These interventions, which we refer to as service mediation interventions, affect additional resources students receive that mediate the impact measured. Evaluations of these types of programs that solely report effects are potentially misleading. Cost-effectiveness analysis clarifies the importance of assessing service-mediated receipt for evaluation purposes. We illustrate our argument empirically from City Connects, a comprehensive student support intervention. We find that the direct costs of the program represent only one-third of the total change in resource use by program participants required to produce impacts. Evaluative statements of service mediation interventions should be accompanied by information on the full costs to achieve effects. Many interventions might be structured in this way and require evaluation that includes an economic perspective.