Kids these days: the quality of new Wikipedia editors over time

Newbie_quality.by_semester.rows.good_faith
The proportion of quality newcomers over time. (2006-2011)

As part of the 2011 Wikimedia Summer of Research, we uncovered a possible correlation between the decline in new active editors that began in 2007 and the rise of warnings issued to new users by bots and automated tools, which started in 2006.
For those of us studying editor trends, the following question has continued to puzzle us: did the change in communications to new users lead to the decline, or can the rise in warnings be explained by a decrease in quality contributions from new users? Perhaps, as some Wikipedians have argued, the new users of today are being reverted and warned more aggressively than those who entered the project in 2001-2006 because their edits are qualitatively worse (e.g., more self-promotional or spammy, less serious and encyclopedic) than those of previous generations of editors.
While the complexity involved in determining what constitutes a “good” contributor to Wikipedia may never allow us to definitively answer this question, our research argues against the theory that today’s newbies just plain suck.
Newbie_rejected.by_semester.rows.good-faith
The proportion of rejection for quality newcomers over time.

To test the hypothesis that new contributors who entered the project in recent years have been more harmful and less interested in positively contributing to the encyclopedia, we randomly sampled the first edits of newcomers to the English Wikipedia from the earliest days of the project to the present. With the help of some experienced Wikipedians, we hand-categorized the edits of 2,100 new users according to a four point quality scale – blatant vandal (obscene language, obvious vandalism), bad faith (jokes and nonsense), good faith poor-quality edit (bad formatting, unreferenced, but trying to add value), and golden (good faith good edits that should not be reverted).
What we found was encouraging: the quality of new editors has not substantially changed since 2006. Moreover, both in the early days of Wikipedia and now, the majority of new editors are not out to obviously harm the encyclopedia (~80 percent), and many of them are leaving valuable contributions to the project in their first editing session (~40 percent). However, the rate of rejection of all good-faith new editors’ first contributions has been rising steadily, and, accordingly, retention rates have fallen. What this means is that while just as many productive contributors enter the project today as in 2006, they are entering an environment that is increasingly challenging, critical, and/or hostile to their work. These latter findings have also been confirmed through previous research.
Newbie_survival.by_semester.rows.good_faith
Survival rate of newcomers over time.

This study has many important implications for community and Wikimedia Foundation efforts to engage and retain new editors. To begin, it reasserts the centrality of one fundamental policy on the project, “Assume good faith.” This research strongly supports efforts in the community and at the Foundation to do a better job of integrating new editors into Wikipedia and its sister projects, not simply for the sake of gaining new editors, but for the quality of these new editors’ contributions overall.
At the Foundation level, this includes major software changes like the creation of a visual editor to lower the technical barrier to entry, as well as more experimental pilot projects like template A/B testing, an attempt to make the template messages received by new users more personalized and clear, and the Teahouse, which gives new users a friendly, low-pressure space to seek help from experienced Wikipedians. With better software and an inviting and supportive atmosphere, the encyclopedia can continue to grow both in quality of material and quantity of dedicated contributors.

  • Find out more about this study at Research:Newcomer quality
  • This work is part of a journal article in submission to a special issue of American Behavioral Scientist on Wiki Research
  • A special thanks to R. Stuart Geiger from UC Berkeley, as well as Maryana Pinchuk, Steven Walling, and Oliver Keyes from the Wikimedia Foundation, for their assistance with this study.

Aaron Halfaker,
Wikimedia Foundation Research Analyst and University of Minnesota PhD candidate

Archive notice: This is an archived post from blog.wikimedia.org, which operated under different editorial and content guidelines than Diff.

18 Comments
Inline Feedbacks
View all comments

[…] has been hectic (you can read about it here and here) Share this:FacebookDiggRedditStumbleUpon This entry was posted in WMF, Work. Bookmark […]

Maybe there would be better retention rates if guidelines like ‘Do not bite the newcomers’ were better enforced.

The statement that “However, the rate of rejection of all good-faith new editors’ first contributions has been rising steadily, and, accordingly, retention rates have fallen” is absolutely untrue. That would have been obvious if someone had actual calculated correlations. For example, the rejection rate for good-faith editors was essentially flat between 2006 and 2010, but the survival rate fell from around .20 to about .05. Moreover, even if there was correlation, that’s not the same as causation. The Foundation seems absolutely committed to the theory that a more welcoming environment will reverse the steadily decreasing number of new editors and… Read more »

I am not a wiki editor but am a passionate follower and as I understand these trends I think one of the reason for this downfall is that wiki has become increasingly popular to students and so called non-nerds or non-geeks or non-web proactive guys. Thus so to say even common man is trying to contribute to wiki but then the standards will obviously come down. Also, increasingly, articles on various not-so-famous topics (like those that may not have been covered by web journals or news reports) are being taken up by new bees and its not their mistake that… Read more »

The common man has no more information to add to Wikipedia and, fortunately, barriers have been lifted against those that add unreliable material. It’s that simple.

Totally agreed on all three of those points. But I think it has become fairly clear, as a 4th element, that we have started rejecting “good faith” edits far more over the last few years, even though the rejection of “golden” edits remains the same. That, to me, is the most interesting thing about the study (although obviously I have a COI).

I remember reading an article somewhere about a notable scientist who had published many papers and won many awards, and was generally regarded as one of the best living scientists in his field not having a wikipedia page. Someone tried to create a wikipedia page about him, and the “notability police” came around and rejected it. It seems wikipedia needs to consider expanding what it considers notable while cracking down on having so many web pages devoted to fictional characters and events. (There are over 9,000 pages containing the phrase “is a fictional character”.) It’s honestly getting difficult to add… Read more »

If I stepped into my university library and six people shouted at me at the top of their lungs that I was not qualified to walk here, that my foot was not in fact attached to my body, and that I should get the hell out, I would never go to the library again.
That also happens to be my summary of my experience submitting a single edit to wikipedia.

Let me translate what I have written in response to changes announcement in an article I contributed: http://pl.wikipedia.org/wiki/Dyskusja_wikipedysty:Naugtur ” Here’s why I gave up contributing. Articles I submited were written in cooperation with a Prof. who specializes in computional linguistics and became a prof doing that, but it didn’t stop an experienced wikipedia editor from “fixing” the article. I also remember seeing an article contributed by a 60yr old MIT professor removed, because some twenty-something experienced editor… Because he thought it was a case of self-advertising. So why would I want to contribute scientific content if anyone can “fix” it… Read more »

x
Wikipedia newbies whine all the time. It usually comes down to their laziness of not wanting to spend the time to add citation references and follow wikipedia rules.
x

Last edit I made (adding a musician to a name’s disambiguation list) was knocked back by a bot. I made a vague effort to follow the recommendations to have the edit accepted, but in the end I gave up.

Established editors who take a proprietarian approach to “their” articles in the face of edits by real world domain experts, editors who speak wiki jargon, the very concept of “deletionism”, these are what is strangling the life out of Wikipedia. Reading the comments above where established editors with stung egos is illustrative.

This is very interesting. I hope more research is conducted that looks into new editor retention. Looking forward to reading the ABS article once published; should be a great issue overall.

I was active in editing for about a year, I’d say. By the end of the year, I’d had enough of the wiki-lawyering, the triumph of process over content, and the death by a thousand reverts to call it quits. Mark Atwood, above, has it exactly right. I see an inward-focused culture, more concerned with their own selves than with their (putative and advertised) goals.

This entire post assumes there’s a problem with Wikipedia. There isn’t. If anything, there’s a problem with the public perception of Wikipedia (or it’s perceived “brand promise”, in marketing speak). Wikipedia is commonly understood to be the website that anyone can edit. There are two big reasons that this is not true anymore: – The vast majority of the low-hanging fruit, in terms of simple edits that beginners can do, has been picked (as commenter Danard said “The common man has no more information to add to Wikipedia…”) – The scale of Wikimedia has necessitated a lot of rules and… Read more »

Dave, thanks for your comment. For an informative glimpse into why we still need more contributors, I urge you to type “Wikipedia:Backlog” into your Wikipedia search bar sometime. You’ll find a sobering view of our supposedly “complete” English Wikipedia. And on the rest of the 280+ projects, the cleanup categories are even more dramatic. Bottom line, there may be a bit of truth to the “low-hanging fruit” theory, but it’s by no means the whole story.

[…] Kids these days: the quality of new Wikipedia editors over time — Wikimedia blog […]

[…] 6. Kids these days: the quality of new Wikipedia editors over time — Wikimedia blog […]