Flagged Revisions: Your questions answered!

There has been a lot of interest lately in Flagged Revisions, a quality control mechanism for MediaWiki. In particular, people want to know when and how that’s getting used on the English language Wikipedia.
I’m William Pietri, a San Francisco software consultant who recently came on part time to do project management for this. In addition to my long experience building web software for on-line communities, I’m also a Wikipedian. Although I haven’t done much more than small corrections lately, I started editing in 2004, and became an admin in 2007.
My job on this has two main parts. The first is to make sure that everybody working on this gets everything they need to make progress. The second is to communicate progress to the wider world. In the spirit of open communication, this is an update in question-and-answer form. Mostly from real questions people have actually asked me, but I’m going to sneak in a couple that I expect somebody will ask shortly.
What is Flagged Revisions?
You can find more detail here, but Flagged Revisions is basically a way to insert a quality review step between someone editing an article and that article version being published for the general public to see. It has been in use on the German Wikipedia since May 2008, and implemented in other languages and projects since then (see this page on Meta for a full list). Typically, in those use cases, every single article is treated in this way, and every change by a new user has to be reviewed.  There are a number of ways Flagged Revs can be used, and the proposed implementation for English Wikipedia (described below) is quite different.
Fundamentally, the objective of this technology is to reduce the exposure of readers both to subtle and not-so-subtle malicious changes in articles (whether it’s the insertion of blatant nonsense, or claiming the death of a celebrity), and to reduce the workload of people patrolling these changes by reducing duplication of effort.
What about the English language Wikipedia?

The use of Flagged Revisions on the English Wikipedia has been under discussion for a long time. Ultimately, the English Wikipedia volunteer community developed a proposal titled “Flagged protection and patrolled revisions“, which garnered strong support. It is fundamentally different from the way the technology has been used so far. Instead of requiring every change by a new or untrusted user to be reviewed, the mechanism would be activated on a per-page basis only, as an alternative to existing tools to restrict editing.
Notably, thousands of articles in the English Wikipedia, typically pages with a very high risk of malicious editing (e.g. major political figures), are currently “semi-protected”, meaning that new or unregistered users cannot make any changes at all. This new tool would make it possible to open up these pages for editing, provided that potentially problematic changes receive positive review. As a result of the more open approach, more high-risk pages could be made subject to this level of community moderation.
Initially, the English Wikipedia volunteer community wants to trial the system for two months. In addition to this alternative to page protection, the proposal calls for implementation of a new feature called “patrolled revisions”, which doesn’t impact what readers see, but is designed to make it easier for change patrollers to organize their work.
How is the Wikimedia Foundation responding to this proposal?

The Wikimedia Foundation (WMF), together with Wikimedia Germany, has driven and funded the development of the FlaggedRevisions technology since 2007 to the point where it has been able to scale to more than a year of production use in our second-largest project, the German language Wikipedia. WMF has carefully reviewed the English Wikipedia proposal, and allocated resources to assess its impact and support implementation along the principles outlined in the proposal.
The technology as proposed markedly differs from the way it’s been used before, so it’s a substantial development and design effort to get it right. For example, the notion of a per-page setting necessitates an entire set of user interface changes that allow changing the setting of a page, and that make it clear to a reader what the state of a particular page is. See this mock-up by Howie Fung as an example of what revised per-page controls could look like. WMF will post further mock-ups for feedback and prototypes for testing as they are built.
Who is currently working on the project?

  • Aaron Schulz, a contract developer with Wikimedia, is the lead developer of FlaggedRevisions.
  • Howie Fung, a contract product manager who also works with Wikimedia’s Usability Initiative, is supporting usability and product review of the software.
  • I, William Pietri, support the project management of the English Wikipedia roll-out as described above.
  • Erik Zachte, Wikimedia’s Data Analyst, will develop metrics specifically assessing the impact of the English Wikipedia rollout.

When will it be done?
This question has been asked a lot lately. Because this isn’t the flipping of a switch but a software development project, answering it requires me to let you in on a secret about software development projects. There are basically four ways to deal with dates, but only three of them are sane:

  1. It’s done when it’s done. Nobody mentions dates. The developers code until they’re finished. Then you release, get feedback, and code some more.
  2. Measure progress and project dates. You lay out all the work, estimate relative size, and then measure how much you get done over time. That data is used to figure out release dates.
  3. Pick a date and release whatever you finish. If you’re building, say, annual tax return software, it’s better to ship on time and drop features than it is to finish late with everything.
  4. Make up dates to please people. This is very popular, and has the advantage of making people happy at first, but it rarely works out well.

Until recently, we were using the first approach. That’s how most Mediawiki development (and most other open source development) works, and it has many advantages. But because a lot of people are eager for this project to launch, we’re shifting to the second approach.
The developer, Aaron Schulz, has estimated all of the items on the work list and already started in on them. The holidays complicate things some, but I expect we’ll have enough data to make a first guess at the estimated release date by the middle of January.
Wait, there’s only one developer on this? Is the Wikimedia Foundation taking this seriously?
Yes, absolutely. Aaron has been working on the Flagged Revisions extension for years, and nobody knows it better. We talked about adding developers, but unfortunately adding more people now wouldn’t help. I haven’t dug into the history much, but it looks like the real slowdown lately wasn’t in the coding; it was in turning the many-voiced community response into a clear set of things to do.
Having spent time with all the people involved, it’s clear to me that the Foundation takes this project very seriously. It’s one of small number of high-priority projects, which include things like keeping the site running, organizing the annual fundraiser, and Wikimedia’s usability initiative.
How can I keep track?
There are a few ways. First, we’ll mention big updates (and the eventual release) here on this blog. Second, keep an eye on the labs site. That will be updated regularly with the latest code and configs. You can judge for yourself how we’re doing, and make sure we do it right. And third, I’ve put the work queue into a public web-based tool called Pivotal Tracker. It’s one of the few software project management tools made for the measure-and-project approach we’re using; if you’re the sort of person who likes way too much detail, you can find real-time updates there.
How can I get involved?
Go to the labs site, play with the current implementation, give feedback, post your own user interface design suggestions, report bugs, and so forth.  Further community discussion in the English Wikipedia about the proposed roll-out is happening on the page about flagged protection and patrolled revisions. I’m always glad to hear feedback, either on my talk page or via email. And of course, you can comment on this very blog post.
William Pietri
Contractor, Wikimedia Foundation

Archive notice: This is an archived post from blog.wikimedia.org, which operated under different editorial and content guidelines than Diff.

19 Comments
Inline Feedbacks
View all comments

Good to see you are making progress. Has flaggedrevs.labs.wikimedia.org been updated with the new code version yet so we can try it and look for bugs?

If Section 230 of the CDA did not exist, this system would have been rolled out two years ago. William, I earnestly appreciate your apparent fortitude and diligence on this issue; but I hope you’ll pardon me for saying “I’ll believe it when I see it.”
This attitude has been shaped after three years of dealing with the duplicity and deflection that the WMF now excels in.

Okay, here’s where I’m a little confused: “It is fundamentally different from the way the technology has been used so far. Instead of requiring every change by a new or untrusted user to be reviewed, the mechanism would be activated on a per-page basis only” Isn’t that really how it always works? Even when you are using it on “every page” it still requires someone to choose a revision to be flagged (because just automatically flagging the current revision would defeat the purpose). And even before the flagged protection proposal, FR included the ability to determine which revision is shown… Read more »

Thanks for the comments, everybody. @Apoc2000: We haven’t yet pushed a new version to labs, but I’m actively working with people to make it happen soon. I also want to make it easier to do pushes, so that we can do them at least every week or two. The more frequent the feedback, the better the result. @Gregory Kohs: Being skeptical about this is reasonable; the status has been murky for a long time. Personally, I’ve seen no duplicity at all; everybody has been straightforward with me, and is eager to be open. That said, I’ve always thought that the… Read more »

I don’t understand how semi-protection will work with flagged revisions. Will all (or most) semi-protected articles be replaced with FR?
Also, how about having all BLPs under FR? Or is that up to the English Wikipedia community to decide?

Gabriel: Semi-protected pages can be replaced with semi-flagged-protection (or whatever it’s called now), but I don’t think all will. Certainly not at first.
I think having all BLPs under FR is a good idea, but it is up to the enwp community. It depends on how many articles we can keep handle, keeping up with approving/reverting all edits.

I know what the difference between the German/English systems is, I just don’t quite see what actually needs to be done to make the software support the enwiki plan, and why that takes so many months.
FR has had the ability to set whether the current or flagged revision is shown to unregistered users on a per-page basis for some time now (predating the flagged protection proposal I believe), and its had a similar sitewide config setting as well.

@William: You said, “I often tell people that if they want maximum openness, they need to create an environment that rewards it, not punishes it.” I couldn’t have said it better. That’s why when I saw the Wikipedia Reward Board offering cash for content creation, I thought nothing wrong of launching my own MyWikiBiz service, where (in the disinfecting sunlight of full disclosure) I would author new Wikipedia content in exchange for a modest fee for my time. Jimmy Wales’ adamant instruction to me was to take it off Wikipedia, post the content on my own site, and let other… Read more »

William, without implying that you are in any way responsible for this, I think that the community is generally a little bit confused about what is being worked on and what will happen when it is done. Part of the problem is using the term “flagged revisions” to refer to several different things (albeit all sharing the same underlying mechanism). It may help to use more specific language in future updates – “Flagged Semi-Protection” instead of just “Flagged Revisions”. The plan seems to be simply to do a two-month trial on a limited set of articles, turn it off again,… Read more »

I’d Just As Soon Remain Anon : The plan seems to be simply to do a two-month trial on a limited set of articles, turn it off again, and “analyse the results”. Surely it would be more expedient to work on one of the proposed options, implement that as a test, and *then* work on the related options (refactoring the original as necessary)? Implementing both “Flagged Protection” and “Flagged Revisions” at the same time, and each with a semi- and full- variant can only lead to a result that will defy analysis (should anyone ever attempt it). Finally, what did… Read more »

Aaron, my mistake – when I wrote “Flagged Revisions” in the paragraph you quoted, I meant to write “Patrolled Revisions”. I’m sure that mistake made my message unnecessarily confusing. Is Patrolled Revisions not being implemented any longer? I’m sorry, I didn’t mean for my post to seem like an attack on you personally, but this has been promised for months and months and months with virtually no visible result. The extension is already up and running in some form on the German Wikipedia, so we’re not talking about an entirely unknown quantity here. It really shouldn’t be this hard to… Read more »

@I’d Just As Soon Remain Anon: We want the limited test to be a fair one. If we put up something that’s hard to use or confusing, a lot of people will have a bad experience for reasons that have nothing to do with the theoretical concept of Flagged Revisions itself. If you look at the feedback we got on the labs site, it’s pretty obvious that we weren’t ready to release yet, which is why we got the usability team involved. As to getting comments based on screenshots, I’m not sure that would be helpful. We need data on… Read more »

@William, all good points, but not realistic. People simply aren’t going to log on to the test wiki unless they have some impetus to do so. I guarantee that if you take my suggestion you will have a much greater number of users doing so and learn a lot more than you would otherwise. Or your money cheerfully refunded. I can’t blame you if you don’t want to have to deal with a new round of arguing about what should be done instead of what is being done, but at least then you would have (a) an excellent reason why… Read more »

@ I’d Just As Soon Remain Anon: I’m perfectly glad to deal with more discussion — in fact, I’m eager for it — but I’d like it to be discussion informed by actual experience, rather than imagined experience. Last time we got a fair bit of actual use, and I hope to increase that with the next round. If that somehow fails, then I’ll definitely look for other avenues, but I don’t expect that to be the case. As to Patrolled Revisions my understanding is that is definitely in the pipeline, but will come after the improvements to Flagged Revisions.… Read more »

I really like the flagged revisions, which I see in action on the German wikipedia, where I am a frequent contributor. But I do believe that they do not go far enough since they always require you to first approve the last edits by IPs before you can work on the article. Thats a huge drawback because sometimes you are not an expert on that field and cannot be sure that the edit is correct. Still if you were to work on the article, the next editor would have to verify YOUR edits and the ones by the IP (since… Read more »

@Hannes Röst: Thanks for mentioning this. You’ve identified a real problem. Your solution is an interesting idea, and one worth talking about down the road. My immediate concern is that branching and merging is notoriously difficult even for trained, paid professionals, and despite all of the effort put into revision control systems and related tools over the years. I’d worry that volunteers would struggle even more, and that the end result would be declines in both contributions and reviews. Of course, that I don’t know how to make something work certainly doesn’t mean that it’s impossible, and it would be… Read more »

William, not to pester you, but it’s mid-January, which is when you said there would be a release of the “estimated release date”. Where are we at?

#17: Great question. As you can see from the tracker, a lot of work is thought to be done. However, for the measure-and-project approach we using, that’s not enough. To be sure things are actually done, we have to be able to release and get feedback from real users, you hopefully included. The surprise delay is that the labs environment is not separate enough from the main cluster; we can’t just push new code to labs without pushing to all WMF projects running on the main cluster. I’m working energetically on getting an environment where we can release quickly and… Read more »

@William Peitri Thanks for your response, I also believe that this might be a big problem, especially since the technically not so inclined already have problems dealing with Wikipedia and its model of version control. I hope we can give it some more thought once this change is out and maybe by then the levitation ppl have something to show for. On the other hand I believe that branching, merging and revision control is only as hard as one chooses to make it and with a good user interface it must not necessarily be a killer for user-friendliness. Especially considering… Read more »