TIPC has set about the crucial and challenging task of developing an approach to the evaluation of transformative policies and experiments that is consistent with its policy principles. This approach is termed the ‘TIPC Methodology which encompasses a Formative Evaluation (FE) Theory of Change (ToC) for Transformative Outcomes (TOs). The FE ToC creation was led by TIPC partner, INGENIO.
Part of this consideration was that, firstly, TIP projects are ambitious and operate within complex environments and systems. It is not only a matter of developing a technology, or deploying it in a specific context. Transformation requires organisational, institutional and cultural changes. Many of these changes take place beyond the location and period of the initial project and intervention. If we wait to assess a policy until its potential impacts are starting to emerge, it may be impossible to attribute such impacts to the policies under assessment. Therefore, the lessons derived from the analysis will be obtained too late to be used in an effective learning process. We need to address attribution and timing problems.
Secondly, evaluation practice is contingent on the nature and scope of the activity to be assessed, the resources and the evaluation capability available, the point in the policy process at which evaluation is introduced, and the governance structure within which the activity under assessment is conducted. TIPC activities are conducted in very different contexts, and involve a wide variety of international experiments across three hubs.
Although we cannot develop a single set of tools for general applicability, TIPC has defined a conceptual framework and a generic approach that can tackle attribution and timing issues that can be adapted to local circumstances.
The main elements of this formative evaluation approach are the following:
- Our evaluation approach is formative and inclusive. Formative evaluation refers to a long-established set of principles and practices that aim to improve policy design and implementation in close collaboration with all agents participating or directly benefitting from these tasks.
- It focuses on the identification and monitoring of transformative outcomes. The Transformative Innovation Policy Consortium (TIPC) is deriving a set of “twelve transformative outcomes” that are traceable and attributable to the policy interventions under consideration, and that can be used as indicators of whether and how the intervention is triggering processes that are likely to be transformative in nature.
- Learning objectives are at the heart of the approach. Monitoring whether and how such outcomes are emerging from the interventions becomes a step in a process of learning that will steer policy development and implementation towards the transformative goals.
- The approach uses flexible “Theories of Change”. The ToC identifies the transformative outcomes that the intervention is seeking and the ways in which the interventions are expected to lead to such outcomes. As the intervention evolves, the lessons learnt from the evaluation process can lead to the ToC to be revisited and adapted.
- It is based on a nested, multi-level approach. Several, interrelated Theories of Change can be used to address different levels of policy activity: projects that operate at niche level, programmes involving a variety of transformative projects, or policy mixes deploying tools across different policy sectors. At different levels of policy activity, we expect the most relevant transformative outcomes pursued to also be different. The design of these ToCs is informed by transitions theory.
