Assessing and measuring systems change: DT workshop

Key takeaways from each session

‍‍

Session 1: Where is transformative change most likely to occur?

Purpose: This session aimed to identify ways to combine the measurement of structural constraints (on major transformative change) with identifying niche innovations in particular locations. 

Takeaways: Niche identification can be automated to a degree through the use of large language models. However, emerging niches may be too vague for a model to identify and defining what to look for can only be done by humans. Constraints to transformative change should be measured at multiple levels, including country-level, but also at more granular levels (cities) and broader levels (geographic groupings).

Session 2: System change vs impact

Purpose: This session aimed to seek ways to connect the measurement of systems change to shifts in system directionality and impacts on sustainability and well-being. The discussion centred on current indicators of impact: asking how these are connected to systems change and, if not, how to build this connection.

Takeaways: Shocks can create the necessary conditions for systems change, but fundamental change requires a change in the underlying rules. The latter starts from change in the shared ideation. Coordinated efforts are needed to facilitate change (such as common standards). There’s work to be done in the finance sector on how to promote long-term thinking and measuring impacts in alternate ways with a clear need for longer-term investment designs.

Session 3: Intervention points

Purpose: This session aimed to review existing attempts to apply and conceptualise intervention points for both single- and multi-system transitions. The team then sought ways to develop the approach further for policy assessment and integrate it with other approaches (such as scenario-building). 

Takeaways: Change should not be coordinated, but facilitated. However, not everyone has the same view on what a green future might look like, and the process of moving to a ‘green’ future can create tensions between individual rights and the common good. When thinking about possible futures, we need to accept that there will be flaws.

 ‍

Session 4: Measuring transformative outcomes

Purpose: This session aimed to take stock of recent thinking about transformative outcomes, in particular how to define these and how they could be used for quantitative and qualitative assessment

Takeaways: There are current challenges such as the continuation of investment in incumbents and the coordination between public and private sectors. Some options could be blended investments and investing in specific regions. There is a clear need for a theory of change.

‍

Research and next steps

It was agreed that there is a need to develop a training programme for people who want to be part of the DT family of projects, so they can become familiar with the framework. The team needs to create a common language, interface, and translation mechanism to support productive research interactions and create a transdisciplinary discourse. 

Many avenues for research activities were agreed on including; continuing the research on industrial modernity, improving methods for regime analysis and setting up a Just Transitions Observatory. 

That sums up our key takeaways from the 3-day workshop, for a detailed view of each session read on.

‍

Session 1: Transformative change

‍

Identifying niches 

When identifying niches the search should start with defining niches. It was clear that humans need to annotate niches and automated classification can check the accuracy of matching this annotation. 

When examining starting points, it was raised that looking at science might not be the most useful for every type of niche, because as an institution it’s very normative. For resilience, a diverse range of approaches is important. 

Lastly, during the discussion of methods, it was noted that before a language model can be applied, concepts need to be defined. But it’s essential to start with qualitative research, as emerging niches can’t be identified quantitatively. However, it’s important to be mindful of how word meanings can change over time. 

Constraints to change 

To create fundamental change, the rules must change first. Despite the fact that large infrastructure systems are difficult to shift, rules always trump infrastructure.  

It’s essential to measure vulnerability to shocks, the institutions related to these, and how particular locations will respond. As well as path plasticity, for early small trigger changes that may push larger system change. 

The operation of governments in specific locations can be a blind spot in current approaches to niche identification and measuring constraints. Country-level data doesn't include cities, which can be incubators and sociotechnical systems operate at different scales (global or regional) and use different resources. Therefore there may be a need for geographic groupings. 

 
Session 2: Indicators for systems change vs. impact

How to capture change in rules

One way to look at system change is to look for shocks. This can be done by analysing different input-output data, and material flows data. It’s important to note, industrial modernity studies show that ideational change comes first: people frame things differently, then it’s institutionalised in a policy, then (hopefully) it becomes practices. 

‍

How are current indicators of impact applied?

As it stands, the concept of planetary boundaries is recognised, but hasn’t yet gained widespread acceptance. The question is: how to implement​​collective action?  

Additionally, measurement practice isn’t always driven by a desire to understand impact, which can lead to unintentional impact-washing. This results in organisations with good intentions, but without the models and methods they use allowing for a meaningful discussion of impact. 

‍

How to coordinate efforts to facilitate change?

One way to coordinate efforts is to create common standards. Resulting in greater impact than individual assessments. Or the possible creation of a new (cross-agency) entity that could hold knowledge production, activities, and information on interventions across the board. An embedded actor could be one potential intermediary. However, for instance, Blackrock would find it difficult to take a leadership role because of its size and influence. 

‍

Measurement and impact 

It was agreed that anticipating the impact of a niche is difficult. However, considering the location, the principles on which it is based, and the actors who promote it will help. Additionally, the group wondered whether shared practices should be developed to achieve a structured change approach. 

 As we know, impact is often spoken of in monetary terms. However, other types of impact could include regional development goals or linking investments with well-being. It was raised that some investors have abandoned the goal of a financial return to invest in social movements. More consideration is required for how the conditions for transformative impact are created rather than simply measuring the already achieved impact or the impact to be achieved. 

In addition to the focus on ROI, investments are still focused on the short term, but sustainability concerns require long-term thinking. One way to address this could be to follow the example of Japanese innovation management with 100-year and 500-year time horizons. Related to this, there is also a need for longer-term investment designs. The Deep Transitions team spoke about creating an investment portfolio product design experiment with three portfolios that each support transition pathways to one of the three DT Desirable Worlds. 

The dIscussion then moved on to highlight that addressing the SDGs one-by-one doesn’t consider the complex interactions that could lead to counterintuitive and unexpected results. A premise of the DT framework is the need to consider interactions, trade-offs, and synergies. 

Session 3: Intervention points (where and when change occurs) 

It was raised that change is often brought about by counter-movements or outliers, rebels, and fringe thinkers. Therefore, it makes sense to replace the concept of coordination with the idea of facilitation as it creates a common ground among willing actors. 

However, there’s also the need to consider crazy and radical futures that explore other options while also being realistic in acknowledging there will be flaws even in transformed societies.  

In Eastern Europe, consumerism is connected to the idea of political freedom, while a green future represents a different kind of freedom. Consumerism is very individualistic, so green futures generate tension between individual rights and the common good.

Session 4: Measuring transformative outcomes

Current challenges were discussed:

1. Currently, almost the entire financial system is still investing in incumbents. How can we reduce our dependence on these big tech companies and build a more open communications system?
2. A fundamental challenge is coordination between the public and private sectors. Public sector investments can be made without expecting a direct return, whereas the private sector tends to be more focused on financial returns. 
3. Pension funds are often not transparent. These are crucial as a large amount of private capital is public money. 
4. Fossil fuels are still hugely profitable. Investors need to find other profitable investments.‍
5. There is a growing recognition that a more open approach to finance is possible. Yet it’s unrealistic to ask investors to invest in transformative projects while still expecting a high return on investment (ROI). ‍

‍

The group considered a variety of ways to stimulate change. ‍

1. ‍Investing in specific regions can create competition, which creates more competition. This approach has the potential to tilt the landscape.
2. There needs to be new relations established between the finance and policy regimes. There could be blended investments, public and private actors together, for transformative change.‍
3. A portfolio-based approach for investors could mean taking complementary actions if they have a stake in different disruptive technologies. A successful niche requires a comprehensive set of supporting functions. This may require a complete rethinking, which in turn could destabilise the current system.
4. Could green funds in pension schemes be a way to achieve both financial and social returns?
5. Could you develop a business case for investing in a region with an ecosystem around a particular system? This would create the elements necessary for overall well-being, with no immediate returns but long-term benefits for the system as a whole.
   ‍

It was clear that both a theory of change and intervention points are needed as one actor (public or private) can’t focus on everything.

‍

Thanks to everyone who presented for these four sessions, they included:

1. Anna-Kati Pahker and Laur Kanger overviewed the development of the Industrial Modernity Index identifying structural constraints, Gaston Heimeriks introduced niche identification using bibliometric data, and lastly, Andres Karjus presented on using large language models powered analytics to measure transformative change.
2. Caetano Penna made a presentation on quantitative indicators for system change, Frank Veraart on well-being and sustainability monitoring, and Jess Daggers on measuring impact in the private sector.
3. Anna-Kati Pahker and Laur Kanger presented on using expert input to identify interventions for Estonian energy and mobility systems, Henrik Larsen on using the intervention points framework to analyse the EU’s Farm to Fork strategy, and Margit Keller on connecting the insights of social practice theories to intervention points.
4. Laur Kanger made a presentation on a unified framework of intervention points and transformative outcomes and Surya Knöbel on measuring transformative outcomes. 

‍

‍