Moving the Needle: How We Test New Ideas Across Wikimedia Projects

The Experiment Platform Team is a relatively new team at the Wikimedia Foundation, tasked with helping other product teams test new features to improve the overall experience with the Wikimedia projects for all audiences. One year in, some new ideas to evolve our software have proved their worth, while others didn’t, which turned out to be just as valuable in terms of learning.

This will be a brief journey across those ideas, and at what structured experimentation has taught us: that knowing what not to build matters as much as shipping what works.

Why we experiment

The scope of experimenting is moving from a “we think this will work” mindset to a “we expect this will work, based on what we observed when we tried it” kind of thinking. We want to make effective changes for all of our audiences, from readers to new contributors to experienced users. In order to do so, it is important to conduct live testing on the projects to see which ideas would improve the experience, without negatively impacting the existing environment for long-time users. While some teams have been experimenting independently for longer, the Experiment Platform standardizes this approach across the Foundation.

Community feedback, input, and idea generation are key, informing us of interest and alignment with the movement. Actually testing product changes allows us to measure if that change improves broad user behaviour. Experimentation positions us for learning: we might believe something will work, and it may be aligned with our values and community interest, but does it actually move the needle?

Experimentation also allows us to test with a limited audience. Wikimedia community members have well-established ways of working, and changes can disrupt them in ways that are hard to undo. Testing with a small proportion of total traffic lets us measure actual effects before making any wider commitments, then deciding whether to ship the new feature, investigate further, or walk away from the idea.

What worked and what didn’t

Paste Check, Reference Check and Tone Check were three experiments we ran in late 2025 on several projects, targeting a common two-sided problem: newcomers often publish edits that do not comply with Wikipedia policies (like adding copyrighted or promotional content) while experienced editors bear the burden of identifying, fixing, and sometimes reverting those edits after the fact. Reversions are discouraging for newer volunteers, and finding and reverting these changes is time-consuming for experienced editors.

Paste Check focused on unwanted copy-pasting of copyrighted texts onto Wikipedia, violating our guidelines on not reusing copyrighted content. Reference Check tackled the addition of long texts to Wikipedia lacking a proper reference to support them, with a dual goal of nudging editors to include references from the start, and reducing the burden on experienced editors who would otherwise need to chase them down. Tone Check, finally, addressed promotional, derogatory, or otherwise subjective language not aligned with Wikipedia policies improperly used by newcomers when adding new text to Wikipedia.

All three experiments were successful: with Paste Check we observed an 18% decrease in the revert rate when the message was shown; with Reference Check, edits were far more likely to add a reference or acknowledge/explain why they did not, when shown the message (around twice as likely on desktop, and roughly 17.5 times more likely on mobile); with Tone Check, we observed an improvement of 6.2% in constructive edit rate overall, and of +14.8% among newer users. Reference Check and Paste Check are now part of the default editing experience for all Wikipedias, while Tone Check is available for all newcomers at the French, Japanese, and Portuguese Wikipedias.

Not all experiments introduce winning experiences, and that’s to be expected. Some experiments conducted on mobile web, such as adding a table of contents or expanding sections by default didn’t move the needle as we had expected, and we decided to shelve those particular ideas. But a negative result is still a welcome answer: it tells us that a particular approach was not the right one, gives us new insight into how our audience behaves, and it saves us from shipping something that would have risked disrupting established flows without actually making improvements for users. Sometimes the most useful thing an experiment can do is tell us when to stop, paving the way for a better future product to be developed and shipped – or to just decide not to go further down that particular road.

What lies in the future

Product teams need to experiment, for all the reasons mentioned so far. Right now, there are new features and structured tasks that are being put to the test to see if they can bring the kind of good change we strive to obtain.

If you want to be informed about what experiments are running and where, or if you want to have your say during the testing, you can follow the Experiment Platform page on Meta-Wiki (and possibly help us translate it into your language!). We are particularly interested in hands-on feedback about what the new features improve, change, or even disrupt in your workflow.