“It was like a roller-coaster in a thunderstorm! Once… familiarity grew, it was easier to talk about concepts. I found the practitioner-academic dynamic so fascinating. I’m so grateful for this experience.”
Evaluation comes at the end, right? Along with the learning? And the outcomes of the project? Not with the TIPC methodology and approach. It does not accept this linear conceptualisation of MEL. The TIPC approach looks at doing things differently – not ‘business-as-usual’. Across the Consortium with policy or programme experimentation, we think again about how to monitor, to evaluate and to learn from projects, overlaying the ‘TIPC methodology’.
Together, researchers and policymakers put policy practice under the microscope to examine and co-create transformative experiments that help tackle the Sustainable Development Goals (SDGs). The newly-devised ‘TIPC Methodology’ is a multi-step learning journey that utilises Formative Evaluation as a springboard towards Transformative Outcomes. Here’s how the South Africa water project team found the new approach.
With the TIPC methodology Monitoring, Evaluation & Learning (MEL) happens continuously in situ, not at the final stages. The MEL strands interweave throughout to make up the strength of the methodology. MEL is not completed by a separate team hovering over, dissecting the project at the end with a tick-box check-list, but by those running, doing, living, adapting and evolving the project, programme or policy. And so, the findings have immediate application and action.
Continuous reflection is woven into the MEL design to inform what next to tweak or nurture towards transformative outcomes. It is key to stress that this cocreation process and space to do this is the essence of methodology and, therefore, its outcome. Resonating from the centre of the MEL design comes the conviction that this continuous assessment, with the necessary fine-tuning of the project, gives the greatest opportunity to build a transformative path. The method is characterised as the ‘Transformative Theory of Change using Formative Evaluation’.
In this final blog for our series, focusing on the experience of the team prototyping the TIPC methodology in the South African Living Catchments Water Provision project, which strives to fulfil Sustainable Development Goal 6, we examine MEL in this context and the implications for Transformative Innovation Policy (TIP) as a whole.
The reflections come from Alex Marsh, Biodiversity & Ecological Infrastructure Policy Advisor and Aimee Ginsburg, the Natural Capital Accounting Project Manager for ecological infrastructure at the South Africa National Biodiversity Institute (SANBI); and Chux Daniels, Research Fellow at TIPC’s coordinating partner at the Science Policy Research Unit (SPRU) at the University of Sussex Business School, and lead for TIPC’s Africa Hub and this project.
The central drive of SANBI is to bring together, for South Africa, the value inherent in both the built infrastructure and the ‘ecological infrastructure’. The first blog in this series discusses this approach further, looking at why it is important to view the built infrastructure as equal to natural ‘infrastructure’. This perspective perfectly illustrates the sociotechnical theory at the centre of Transformative Innovation Policy (TIP).
Ginsburg works to help manage and guide development of natural capital accounts to demonstrate in economic terms the value of ecology and natural accounting for the nation and the world. She looks at how this translates for the Sustainable Development Goals (SDGs). Her role is to support the decision-making and direction of work with evidence and data centred on natural accounting. As with the Living Catchments Water Provision project, natural accounting is done for each new project. The organisation of information about the environment is key to tackling climate change and biodiversity loss. For example, Ginsburg’s role is to audit the South African eco-systems and landscapes to understand how, for instance, they may have been altered for other land uses effecting biodiversity. As is achieved with other global metrics such as Gross Domestic Product (GDP), how can standardisation take place so there are consistent and transparent ways for measurement? How can this data support MEL indicators and metrics in experiments and projects?
This global standardisation for understanding the value of natural accounts has recently made a leap forward with the newly released System of Environmental-Economic Accounting (SEEA) framework from the United Nations. This integrates economic and environmental data to give a better understanding of the connections between the economy and the environment, highlighting the value ecosystems bring to countries and their people. MEL activities will need to factor this in alongside the SDG framework. For MEL within the TIPC methodology – an explanation of which is detailed in blog 2 of this series – Ginsburg has a multitude of pertinent questions:
“Where are the data limitations? How can it draw out indicators for SDGs? How do we communicate the natural accounts effectively? How do we use them? What are the policy implications?”
Ginsburg sees potential in expanding the TIPC methodology across a broad range of programmes and explains her motivation and interest in this new methodology:
“We are developing the National Strategy on natural accounting, and I would like to apply the TIPC methodology to this development process. What other ways can we adapt to build in what we are learning? This is why I’m interested in TIP, and being part of the experiment to integrate the TIPC methodology into to our Living Catchments project.”
The second contributor, Alex Marsh, describes how she works for SANBI resource mobilisation working strategically on lead components involving: practise for policy advice and mainstreaming; ways of working with people and institutions to bring science-based policy advice to this. This is her involvement in the project: “I am involved in component three of SANBI’s Living Catchments Water Provision project which examines public and policy processes.”
Marsh’s initial impression of TIP involved a ‘deep curiosity’. She explains: “It was great to come across people thinking in these ways. SANBI has been a trailblazer in mainstreaming… for fifteen years. However, we are relatively unknown in academic circles as we don’t write papers. So, interaction was great; about how research can work meaningfully in the practitioners’ space. How can we cross-pollinate? Social process is key. How do we allow experiences to change how we are working? In the public sector the pace is intense. Ongoing relationships with TIPC helps us to knead in what we are learning.”
The discussions began by centring on the complexity of MEL and this fresh perspective – how have the SANBI’s team conception of indicators changed? What’s been learnt here?
Firstly, for the process, Marsh suggests:
“I think we could have had an extra session on designing indicators. The indicators are very pragmatic and linked to outcomes in the project. This is about finding ways to shift practice. It’s very difficult to measure impact and outcome in a meaningful way. It has to be quantitative and qualitative. Stories are important to demonstrate the shifts brought about by implementing the project; to speak from experience and to not be reductionist.”
“I had to manage expectations around what could be an ‘outcome’. To try and shift and measure people’s behaviour out of new information to show learning in a co-learning modality. The knowledge products can be done in a deeply collaborative way. Producing the materials with the hardwired engagement put into the process is part of the mainstreaming work I am doing. I have interviewed people and will go back to do this as part of the MEL.”
In reflecting on cocreation as part of the project, Marsh captures the experience of the team:
“I have strong opinions on this! The first meeting was like a roller-coaster in a thunderstorm! I didn’t understand the terrain I was in and then we were rolling. The principle to building trust is to allow collaboration. Perhaps, the TIPC team could think about how to soften the entry to find ways to have conversations prior to, to help us all find each other in a more meaningful way. Once that familiarity grew, it was much easier to talk about concepts, as you knew where people were coming from. The practitioner-academic dynamic I found so fascinating. I’m so grateful for this experience.”
“We talk a lot (about the transformative perspective), reflecting on it in SANBI, we are a bunch of practitioners’ scientists, we collaborate with academics but we feel the difference between approaches very intensely and I think that exploring the skills sets in the room is important.”
“What SANBI does best in the world is operate in a niche about using deep social process and being able to hold space in a way that allows people to feel like they have voice in the process.”
Providing a recommendation for the cocreation process to blossom further, Marsh concludes: “If we had paired our skills sets prior to the programme of work, we could have framed the conversations better. For example, our deep social processes working with those of the transformative outcomes. For cocreation this would help to be explicit about our hopes. This would have added another layer of magic!”
Ginsburg supported this observation explaining: “Yes, in the first few sessions it was hard to locate myself, even though, I did the reading and ‘got’ the theory – it was the how? How does it all fit together? Some sort of pairing of the TIPC process and the social learning process (that we use) could have helped to give space for finding each other.”
This suggests some analysis and mapping of the interrelated process used by the TIPC experiment teams, prior to or in the early stages of the prototype project, would be advantageous to the MEL development, particularly for the conception stage. This would allow people to ‘locate themselves’ and have understanding of the context and discourse that each team are embedded in. Consequently, this does emerge through the process, although emphasis on this in the ‘onboarding’ stage is highlighted as a tactic to help foster the positive results more swiftly with early establishment of trust, awareness and rapport.
This would help with the team dynamic and the initialisation of the projects. Here the approach moves further away from the jump straight in, ‘business-as-usual’ approach where teams may not be ‘formed’ in a nuanced, ultimately more productive manner, by paying close attention to the human interactions at the centre. For this to succeed, both a time commitment and momentum is required. With the South African water experiment as this was the first time the methodology had been tested, the TIPC team was also still learning. It is an experiment on both sides.
From the TIPC researcher side, Chux Daniels, lead for the TIP Africa Hub, reflected that: “Not only at the beginning should there have been more time but also for the MEL session themselves afterwards”. Daniels advises two more sessions be added for MEL in the concluding component of the TIPC Methodology that guides each policy experiment. The additional sessions will focus on the development of indicators, in addition to providing space for testing and reflections on the selected indicators. Contemplating this further, he identifies that the tension and balance is between: available time resource; keeping the necessary momentum and structure; and amply exploring the MEL experiences of the teams. Referencing these, Daniels finishes: “There were many factors shaping it all. These are the lessons we’ve taken on board to pay attention to. And all this was within a time of COVID adding to the challenge!”
A further positive aspect covered in the interview were ‘reflections on the reflection’. In producing the South Africa Water Experiment Blog Series, to provide the opportunity to illustrate the experience and viewpoints of the teams, we created an important space for this essential reflexive activity that has been appreciated by those involved. Ginsburg explicitly raised this point at the end of the interview: “I’d like to add how important this is. This process of interviewing and reflecting after. We have talked about the ‘onboarding’. For me, this ‘outboarding’ debrief too is really helpful for the next stage of work, particularly for how it all links to the new South African National Strategy that I am working on. Reminders and pointers are coming to me from this discussion. It’s an important part of the process.”
Marsh continues: “Another point linked to the value of substantial debriefing is that we were all surprised by the level of intensity. As a team, we did another deep reflexive exercise about how it was going for everyone, and this was really useful for us. So, the advice would be for the policy teams to build this into the side of the main programme. This session was, I have to say, quite beautiful.”
As well as focusing down into the team, Marsh and Ginsburg observed that by sharing the team’s experience with a wider audience, in this instance at the regional learning event in South Africa, which took place on 20-22 October 2020; there was a lot of positive MEL aspects.
“The session in the wider South Africa forum, where our colleague, Shanna Nienaber talked about the programme to a wider audience, this opened up the space for the important conversations around – what actually happened?!”
Demonstrating the long-term perspective being taken on the TIPC work and the positive relationships that have been developed, Marsh concludes:
“This debrief process is so great too as we know we have got years together…!”
In rounding up the reflections on the team’s learnings, Ginsburg shows how the experience will inform other important areas of work in South Africa:
“I want to more clearly develop a (transformative) theory of change with other partners as it a valuable exercise. It has bought more factors to my attention as the strategy is developing. Our social process stuff will be the nuts and bolts of the work, but what is important now is how it’s undertaken, and what it will achieve. It’s the softer elements. Cocreation will now be interpreted differently. We should think about how we articulate the TIPC methodology, and how to use it with other partners such as Statistics South Africa.”
Marsh enthusiastically puts her final thoughts across:
“Having new colleagues to think about how you work on strategy and policy is the greatest possibility that has come from this! This emerging partnership is great. To have a field of experts to think with and have a community of practice for this niche of work. We have been thinking about theories of change for years – about how to do them?! The government is saying they want more theories of change, but not how to, so this is a very alive and generative area to take forward. Secondly, with this Transformative Theory of Change, I’ve had conversations with partners who are interested in what we have done, who are watching what comes out of it, we are looking to apply this to the programme and process design on bigger projects. This is sitting on the horizon. This framework could be powerfully applied. We would like to write with the team on research papers too. I know how valuable this would be to do with the TIPC team. For policy and research, to knit to these together to make our work more meaningful, to bring about the change we all want to see in the world.”
Top Takeaways on Conducting a TIP Experiment:
For Policy Teams:
- Soften into the theory with more activities and time to build trust and rapport with the TIPC project team so that the theory lands more easily and people can ‘locate’ themselves within it. At the beginning it can feel like ‘a roller-coaster in a thunderstorm!’ Prepare for the change in perspective and not doing ‘business-as-usual’.
- Trust and good collaboration arose midway in the process. But at the beginning, as with many things, it was daunting and there was uncertainty. So, be open to this and have reflexive sessions for the team built into the side of the project.
- The prototyping experience and learning can be taken into other key projects. For example, South Africa are applying it to their National Strategy for Natural Capital Accounting (NCA).
For TIPC research teams:
- For Stakeholder management, have committed and deeper conversations and workshops with partners to discuss their needs, mutual expectations of the collaboration, their perceptions of TIPC, and experiences with MEL throughout the process.
- Clarify the role of the TIPC team (not consultants, but partners) and the purpose of MEL for cocreation towards transformative outcomes.
- Dedicate more time to translating concepts, brokering activities, and overall accessibility. Making a reiterating connection to project partners’ work and contexts.
- Translate theoretical concepts into something meaningful to partners, emphasize a two-way learning process.
- Add an internal ‘team learning’ strand to the learning strand.
- Involve the same people across various policy experiments to support learning and facilitate the transfer of knowledge and understandings.
- Use and understand theoretical concepts in similar ways across projects, especially since generalization helps identify common challenges in different contexts.
- Develop generalizable tools for beyond the TIPC lifetime.
What a wonderful story and experience! Well done to TIPC and the team ?