One of the greatest challenges of project work is making sure that people understand the intended long-term outcomes of a project and how they can be achieved. This is especially true for organizations such as Wikimedia Deutschland, which works on a number of interconnected projects at the same time and therefore has to deal with a corresponding level of complexity. Even people within the organization can quickly lose track, wasting attractive opportunities for collaboration in the process. Most importantly, the great complexity of various isolated projects and goals makes it very difficult for organizations to establish a shared pool of knowledge and to increase the effectiveness of their actions in the long term. At worst, different organizational units can end up working at cross-purposes without even realizing it.
Wikimedia Deutschland believes it is important to avoid problems of this kind, not least in order to be able to evaluate the impact of our own work meaningfully. The most important question for us to ask is whether we are achieving our societal goals. In order to develop useful indicators and metrics to answer this question, you have to first be unambiguous in what you want to achieve in the long run. For this reason, the evaluation unit was appointed to work together with our employees on developing change models for the work we do at Wikimedia Deutschland. Here, we were able to build on the foundations laid by the Program Evaluation & Design Team at the Wikimedia Foundation (thank you for the good work, Jaime!). Since various parties (including the FDC) have recognized and expressed an interest in our change models, we would like to briefly explain what change models are, why we chose to use them and how we went about it.
What is a change model?
Change models determine the direction of impact for potential projects and serve as a theoretical basis for planning and subsequently, evaluating projects. Ideally, a change model will be drawn up before the conception phase of a project. It always pursues a specific intended long-term outcome and explains how this goal is to be achieved through the interplay of several sub-outcomes (preconditions). In addition to outcomes and sub-outcomes, a change model can also include measurement indicators, activities and other information (see Theory of Change, for an example).
What do we hope to gain? Openness and sustainability
Openness: Change models help to convey what Wikimedia Deutschland wants to achieve in the long run and how.
Wikimedia Deutschland not only deals with a number of interconnected outcomes and projects, but also works with many very different stakeholders with varying requirements. These include the active community, our members, WMF, Chapters, FDC, donors, project partners, experts, project managers, department heads, the executive director and of course our supervisory board.
Change models help everyone involved in our projects to be on the same page about what we want to achieve in the long run and how: It is often necessary to explain our intentions to each specific target group individually. For example, change models for grant-making and evaluation are typically visualized using complex flowcharts in which arrows indicate the dependencies between the individual intermediate goals. However, an explanatory text may prove more suitable for other target groups. With change models at the center of our work, we have a lower risk of losing sight of the overarching storyline.
Change models therefore enable outcomes and projects to be communicated in a way that is tailored to the relevant target group but always has the same content. This makes it easier for interested parties to identify specific links with their own ideas and projects. One goal of the annual planning for 2014 was to remain as open as possible to the ideas and projects of third parties while being able to respond flexibly to new social legal, and societal circumstances.
Sustainability: Change models create a long-term common frame of reference.
An organization’s strategic priorities and operational planning can change quickly in response to adjustments in internal and external circumstances, which means that they have limited suitability as a long-term common frame of reference. A more appropriate basis would be the underlying causal outcome assumptions that can be mapped in change models. Although these are not set in stone, they tend to change far less rapidly. Each individual outcome assumption can be systematically examined through research and evaluation. This is something that we can all do together, project employees and interested community members included, of course. Should an outcome assumption prove to be incorrect, this will logically cast doubt on all plans derived from it. And so it should provided that these findings lead to the gradual optimization of the change models and the associated plans.
Thus, change models are not just the basis for project planning but also for annual and long-term strategic planning. They can help to ensure that projects do not become side-tracked by minor short-term successes and mere improvements in efficiency. Short-term successes may be easier to achieve and measure, but it is too easy to lose sight of larger potential successes that can only be achieved in the long term.
How did we proceed?
We knew that the greatest challenge in annual planning for 2014 would be to establish a consensus between all employees, the executive director and our supervising board. In 2013, 22 individual projects mostly working independently of one another, a rudimentary long-term strategy and changes in personnel had resulted in divided objectives.
Our first step was therefore to hold numerous identity-focusing workshops to hone the central long-term impact assumptions upon which our work had, up to then, been implicitly based. The first intermediate result under the guidance of the evaluation unit was to form cross-department programs comprising all existing projects at the organization that are using the same basic mechanism to achieve the same long-term outcome.
A change model was developed for each of these programs in further joint workshops. The change models aim to explain the central long-term outcome of a program and how it can be achieved in steps that build upon each other. The change models were developed step by step on several levels ranging from the central long-term outcome through to medium- and short-term outcomes. In preparation for evaluating outcomes, preliminary indicators and activities were then developed for each change model (see Volunteer Support Program, for example).
The change models – including the indicators – were published by the evaluation unit at the same time as the annual plan for 2014. From now on, the unit will continuously check the models on the basis of recognized standards and develop them for the long term. On the basis of the change models and indicators and with the support of the communication unit, the departments then developed the specific targets and activities for the 2014 annual plan. The executive director and department heads took these on a roadshow around Germany to answer the questions of local Wikimedians.
At this point, we would like to communicate relevant findings from the practical experience we have gained from our process, for which preparations began in the middle of the year through training sessions and practical methodology exercises.
- Continuity of personnel: Part of the process took place during Wikimania. This is why the group of people involved in the process has changed repeatedly, including key personnel. For this reason, numerous process steps had to be explained again to the new people, and some decisions already made by the group had to be reopened. These unplanned changes in involved personnel have caused significant disruption and will be reduced in the future
- Early involvement of stakeholders: Implementing new change models is a complex business and the time is “never” right to fit such a time-intensive process into one’s day-to-day work. However, the efforts pay off in the long run because there is no need to constantly reinvent the wheel and, as described above, you can gradually build upon insights gained. Although the method would have expressly permitted us to do so, a lack of time prevented us from involving our community and experts to the extent originally planned. Anyone planning such a process themselves should ensure that they have plenty of lead time and that all participants are involved at an early stage.
- Heated discussions are a good thing: We at the evaluation unit were surprised by just how differently our stated goals were interpreted within our organization. It was also surprising to see how long these differences of opinion remained subliminally within the process before an existing consensus was actually accepted by all participants. We therefore decided to tackle such differing visions and interpretations of outcomes head-on in open groups as soon as they became apparent. Involving various stakeholders in the discussions slowed the process. It was important that we use this as a way of really confronting key issues in order to achieve a true common understanding and boost identification with the long-term outcomes, programs and change models.
Introducing change models to Wikimedia Deutschland, we were faced with a major challenge that we tackled in several iterative cycles. This involved a serious investment for all participants in addition to their daily work, but the results are impressive. With the change models, we have laid the foundation for sustainable change and established a practical basis for planning and evaluating our projects. There has already been a notable shift within the organization into the outcome-oriented mindset. Thanks to everyone who have played their part!
Change models depend on continuous development. And everyone can get involved by joining with the rest of the Wikimedia movement to question and examine the underlying assumptions. We are convinced that change models are a central tool to help the Wikimedia movement bring about the greatest possible societal change. We will keep you updated about our progress.
We would be delighted to hear your ideas and questions: Take a look at our evaluation portal on Meta-Wiki or reach us at email@example.com.
Manuel Merz, Head of Evaluation Unit, Wikimedia Deutschland