Tensions and Pivots: Evaluating the Energy Futures Lab

Accelerating the transition to a sustainable energy system is a pretty big goal. Let’s face it, if we could meet this challenge with a step-by-step approach, we would have solved it by now.

But we know these change processes  are highly complex and very difficult to predict in advance. In addition, the EFL has many moving parts – Fellows, workshops, prototypes, public events, partnerships with businesses and governments and all of the activities the Fellows are doing in their day jobs.

How do we keep track?

How can we make sure that we are delivering value to the Fellows and contributing to systems change?

How do we know when we getting to the “breakthrough results” we are looking for?

From the beginning, we made a serious commitment to evaluation, specifically Developmental Evaluation (DE).

In contrast to summative or formative evaluation, developmental evaluation is used to track changes in a project in response to events on the ground. It is also designed to help people running the project to adjust and pivot on the fly rather than waiting for a couple of years to get feedback that something is not working well.

Mark Cabaj has been leading the evaluation work for the EFL and I have been collaborating with him from a research lens. We have been collecting data at a number of levels. First, we have detailed feedback after each workshop that help surface issues to address and areas we can explore further in future workshops. We have also conducted in-depth interviews with each of the Fellows before the Lab started and again in April.

Figure 1: From “A Developmental Evaluation Primer”

We ask questions about what changes are happening with Fellows as individuals, in their prototype, working teams, their organizations and their social/professional networks. We uncover  barriers to, and opportunities for, transition and where Fellows are seeing signs of transition in the energy system. These techniques give great value for both theory and practice. From a practical side, the results of the feedback and interviews leads directly into the EFL design. These DE process are providing the Design Team with the information they need to be highly responsive and ensure Fellows are receiving the support they need to tackle the challenges inherent in their prototype and working groups.

From a theory side, my PhD research is on the societal effect of processes like the EFL – what effect do they have on policy, regulation, organizations, narratives of energy in Alberta, attitudes and behaviour changes? These are big questions and we are still early in the process.

So what did we find?

First, leading up to the last workshop in Kananaskis we found a set of tensions. As you may expect from such a diverse set of Fellows, there was a variety of feedback. Some Fellows felt we were spending too much time on process while others valued the learning about new forms of facilitation and feedback. Some felt the prototype teams were not making progress, others were very enthusiastic about their work.

What was common across the feedback was a set of 3 pivots. Fellows signalled that they were now prepared to begin:

1. Transitioning from one-off prototypes to working groups with a portfolio of initiatives to build together
2. Advancing the learnings from the organizational engagement pilot with Suncor to create more structured engagements and tools to share the EFL experience with more people and organizations
3. Shifting from Fellows working inside the EFL to the EFL having a public voice

Anticipating the significant steps forward, we designed the latest workshop to include more time to think bigger about how EFL Fellows are going to work with partners outside the Lab. In preparation, the Fellows developed the Vision and Innovation Pathways document to share across Alberta, and began to self-organize into working groups based on the Innovation Pathways. More to come soon on the Innovation Pathways!

What’s next?

We are continuing to track progress of the prototype and Innovation Pathways groups and will be interviewing Fellows again in the Fall of 2016 and twice more in 2017 to see what changes are happening. We will also be digging deeper into tracking how the EFL is creating impact through evaluation methods such as Bellwether interviews, process tracking and media scans as part of my thesis research.

A big part of our research goals is to continue to bridge the gap between practitioners and academics. People doing this work on the ground struggle to integrate often abstract theoretical research while researchers often miss the amazing work that is happening in real-world projects. The EFL is bridging this gap and using research to both inform the design of the EFL and to share our learnings with Labs practitioners in Alberta, across Canada and around the world. I’ll be coming back every few months to give an update on our evaluation strategy so stay tuned.