On the value of experimentation – A Future Audiences perspective

Johan Jönsson works as a manager at the Movement Communications team, supporting Product and Technology work at the Wikimedia Foundation. Among other duties, he’s part of the Future Audience initiative. He’s been part of the Wikimedia movement since he got sucked into fixing Wikipedia typos in 2004.

The future is a strange country. All roads lead there, but some are better than others.

In one of the influential classics of product strategy theory, The Innovator’s Dilemma, Clayton Christensen investigated how technologies are overtaken by new ones, and how the dominant organisations struggled to retain their positions as their industries shifted to a new paradigm. Christensen, a scholar of business administration, pointed to how leading companies found it difficult to adapt to the situation even when they wanted to direct resources towards innovation in the new area. It’s difficult for a leading organisation to get truly excited about small, incremental victories when they are dwarfed by the number of people making use of the currently dominating, successful thing the organisation does. And companies have customers, who want their needs met. The pressure from the existing customer base anchors the companies in the old technology, until it’s too late, and the customers have moved on to the new solutions – developed by someone else.

The Wikimedia movement isn’t a company and Wikipedia and its sister wikis aren’t commercial products, but there are still similarities, just as there are Wikimedia-specific factors which make it difficult for the Wikimedia Foundation to try new things. There’s always the discussion about how to handle limited resources, both within the Foundation and within the broader Wikimedia movement, where evolving our platform draws on already overstretched systems, but beyond that there’s the crushing weight of how we relate to each other within the movement.

There are expectations of Wikimedia development. Wikimedia editors like to know what the Foundation is trying to achieve and be part of the development process at a reasonably early stage. It’s not that Wikimedians don’t want us to try out new things as much as the community pressures inherent in the system push development towards clearly defined projects. Editors tend to look at Wikimedia development with certain questions in mind: How can I or my home community use this to add or curate knowledge? How will it fit into the current workflows? How will it affect how readers interact with the information we supply? 

One of the ways Christensen has identified in which companies can get around the pressures and expectations of their success is by having a separate organisation to drive new, more experimental features. A group which doesn’t try to appeal to the same customers, who aren’t bound by the same logic, who can celebrate small victories. This is very far from what the Wikimedia Foundation does, but there is an initiative which is explicitly all about experimentation, and being part of it is a bit of fresh air after nine years at the Foundation.

Future Audiences is a small team, with limited resources but also with none of the expectations of finished products. Future Audiences doesn’t do products. It doesn’t have the resources to maintain something – if something turned out to be successful, and we’d want to keep it, the only product Future Audiences can deliver is the result of the experiment and a recommendation that some other part of the Wikimedia Foundation invest in it.

The main point of building a ChatGPT plugin wasn’t to get a successful plugin. It was to understand how people interact with Wikipedia and information in ChatGPT to see how it could affect the Wikimedia wikis. The point of building a Chrome extension like Citation Needed to see if people can use it to see if a statement is supported by Wikipedia isn’t to build a successful Chrome extension – if it had turned out that this is a product we want, it wouldn’t necessarily have been a browser extension at all – but to see if this is a feasible way of letting users interact with information and Wikipedia.

As of writing, the Future Audiences team is working on a new tool to help readers add facts to Wikipedia – to see if we can lower the threshold to add something by using generative AI to help find the right article to update when you see a fact somewhere and make it easier to add a reference. We’re not trying to use generative AI to create information, but to function as a bridge between the source and the wiki. It’s merely a tool for the human editor.

Yet again: The experiment aims to be quick and cheap. And so we take what shortcuts we can. Initially, it will post updates to talk pages, to suggest edits. And it won’t work – when we launch it, at least – in languages other than English. Were this a product, we couldn’t and wouldn’t build it like this. But it isn’t. We hope Wikimedians would find it useful, but more than anything, it’s a way for us to learn so we can help the rest of the Wikimedia movement understand how the technology works and could work for us.

Wikimedia technical development has historically not focused on small experiments, at testing something for a couple of months and then walking away from it, no major investments. Having a separate function, working according to a different narrative, which can test something and walk away from it? It’s a deliberate attempt to give us the tools and the understanding we need, as a movement, to meet the future. In working with the Wikimedia movement, a significant part of the challenge is to explain what we are doing – that we’re often closer to research projects than products. It’s easy to start thinking about the tools we develop as products, but that leads us down the wrong path, because if we build products we need to invest significant resources to make sure they work for everyone. And then we won’t be able to learn, to be able to add more knowledge to how the changing technical landscape might affect us and what we can do about it.

We think we can make the future better, but only if we’re part of it.