The Rise of Warnings to New Editors on English Wikipedia

(The following is the second installment in the Community Department’s series on preparations for our summer research into Wikipedia. As with the last post, keep in mind that these numbers and findings are preliminary – at this stage we are only testing out different methods and practices. Over the next few months, we will work on scaling each of these experiments and continue to probe more intensely into their methods and results.)
In preparing for this summer’s research projects, we have been experimenting with qualifying and classifying comments left on user talk pages, a space where editors can discuss and receive feedback about their contribution to the encyclopedia. In the process, we began to see a troubling trend that appeared at all stages of our sampling.
We prepared a random sample of several hundred edits made to user talk pages of new registered users on English Wikipedia from 2004 to 2011. These edits were made by other contributors within 30 days of a new person’s first edit. We then tagged each edit with several labels and noted the tone of each message. We excluded talk pages of vandals or users who were banned for spam or abuse of multiple accounts – in other words, we tried to include only talk pages of new editors who were clearly trying to participate in good faith.
When we began to add up the totals per year, the results were striking. Again, we want to strongly caution that the purpose of this exercise was to experiment with methodologies for sampling of Wikipedia data – not an end in and of itself.
Nevertheless, when we analyzed this sample, we discovered a distinct trend: a marked decrease in praise for contributions (anything from a simple “great job on that article!” to a barnstar), and a simultaneous increase in warnings and criticism delivered via templates (e.g., third- and fourth-level vandalism warnings, copyright violation warnings with aggressive images like stop signs or red X marks, and threats of block or bans) since 2006. Note that we excluded warning templates from this group if they were extremely gentle (e.g. “Welcome to Wikipedia. Your test has been reverted, please use the Sandbox…” etc.)

Praise_versus_Negative_templates,_English_Wikipedia_2004-2011

As you can see from this graph, in 2011 almost 40% of all initial edits to new user talk pages in our sample were negative templates, while praise was virtually nonexistent. Even accounting for anomalies within individual years of our samples, the trend suggested is clear.
In our last blog post, we pointed out that while vandalism has been on the rise in the past five or so years of the project, it appeared that good-faith editors still represent over half of all new registered contributors – a trend that we found in this sample as well. So while the increasing use of negative template warnings may be a reaction to increased vandalism on English Wikipedia, it is possible that large numbers of good-faith editors are getting caught in the crossfire.
Over the next few weeks, we will be looking at ways to fine-tune our system by getting more involvement from the Wikipedian community. If you would like to find out more about our classification system or join the experiment, please sign up on our project page on Meta. Documentation of our process and data is included there as well.
Maryana Pinchuk, Research Fellow, on behalf of the Community Department

Archive notice: This is an archived post from blog.wikimedia.org, which operated under different editorial and content guidelines than Diff.

15 Comments
Inline Feedbacks
View all comments

I think you meant “Fraction” on the y-axis rather than “Percent”. But even then: what about the remaining >60% of comments? Were they ‘gentle’ ones?

Hi Mike, We actually did mean percent. I know the numerals on the y-axis are slightly confusing because we didn’t label them as percentages, but our charting program failed to let us label 40% as that instead of .40. Sorry for any confusion! To answer your other question: we wanted to save the deeper analysis of the rest for another time to make this point, but if you’d like to see the raw data let me know. A few other trends I can describe though: the majority of all interactions in all years were either warnings, tips, or corrections of… Read more »

Interesting work. It seems it’s going to be a really interesting Summer for Wikimedia projects. Two comments: 1. I’m curious about the approach followed to label emotional traits of comments in this sample (that is, to classify them as positive or negative feedback). As you suggest above, some of the apparently negative comments may be related to assessment about policies or guidelines that new editors should follow (again, suggesting the need to create a clearer path for new editors to discover and learn them before editing). 2. I suspect that, as a result of development efforts to be conducted next… Read more »

Hi Felipe, Thanks for your very thoughtful comments! Responding to your points… 1. Does it make it clearer to point out we classified type and tone separately? So if there were feedback comments of any kind, we noted that regardless of tone. We then noted afterwards the tone of every edit in the sample. So our dataset was structured in a way where we can see type or tone individually, as a whole or associated with a certain kind of talk page edit. We specifically did that because you’re correct, negativity of tone definitely doesn’t mean commentary was not focused… Read more »

The trend clearly shows a problem. Where is the discussion about how to fix it?

DFB, While I’d caution once more that our sampling at this point is far from scientific, anecdotal evidence does suggest that this graph reflects a major problem on English Wikipedia – an increasingly hostile environment for new users. Discussion of how to fix it is on-going. Some work has been done by the Wikimedia Foundation Board to get the discussion going (see their latest resolution on openness in the community). In English Wikipedia, this has led to the creation of the WikiGuides project to protect and nurture good new editors, and in the Russian Wikipedia, there is a similar pilot… Read more »

I can think of a few possibilities: * new users today are worse than those in the past – perhaps because those most likely to thrive at Wikipedia might have been attracted early, because they are contributing for less altruistic reasons, or because the page rank means everyone and their 8-year-old ends up on the site * new users are less likely to have a positive impact because the quality of existing articles is higher than in the past * new users are just as bad as in the past, they’re just getting caught earlier (before they get any better)… Read more »

GreenReaper,
When it comes to the question of how good the first edits by these newbies are, we did some research that suggests there are just as many people making quality edits that are worth being praised instead of being warned. (Even accounting for the higher editorial standards for the encyclopedia.)

For someone who is contributing at an irregular pace but still a few times through years and having started a few pages. My experience varies depending on the topic, but it is happening more often that the first edits of a page trigger reproaches when it is still in the birth moment of the page. Some editors seem to want to more complete first page, which is cutting the good faith and will and even desires to continue. Maybe an intermediate solution could be found. The page having a kind of draft stage for a little while and instead of… Read more »

Do you have any data on what percentage of the negative messages were due to the new editor creating an article that was deleted? (“An article that you created has been tagged for speedy deletion”, etc.) That might have relevance for the current discussion on restricting article creation to autoconfirmed editors.

That may be the case, but who has time to praise them? I have to spend all my time trying to fend off deletionists from articles in “my” WikiProject! 😉 Perhaps it was done in the past by recent changes reviewers, but from what I can see those people have been replaced by bots and Twinkle cyborgs – in part because almost a quarter of new creators’ edits are vandalism, and serving on the front lines is a clear route to the mop. Now the decision is not “What should the response be to this editor?” but “Is this edit… Read more »

Dana,
We did see speedy deletion templates cropping up more and more after 2007, and those definitely tend to be very negative (big red icons, bolded text, etc.), but we didn’t keep track of the exact numbers. We’ll be sure to flag this question for further study.

This is very interesting data! I can confirm the “anecdotal evidence” as noted in the comments. I do a few edits per year, and get notified of older edits / articles I’ve contributed to. All reactions / notifications are negative. Also I see, slightly related, a change in extreme measures when something doesn’t meet the standards. I see it more and more that (old/long) articles get plain deleted because they don’t meet the standards. Instead of working together to let them meet the standards. This is of course not the same as comments on user pages, but to me it… Read more »

Lode,
Thanks for the feedback! We hear stories like this all the time, and now we’re finally starting to collect the numbers to back them up.
An update about where to discuss this issue: here is a good place to start.

This posting is right on the money. A few years ago I became interested in contributing to WikiPedia and full of enthusiasm I began by adding an image of a camper trailer I designed and built along with a sentence of text. Initially my experience was good–someone came in and adjusted the image size and then another found a more relevant area to place the content I authored. Unfortunately, after that initial good experience the editorial zealots took over and issued a show cause notice why my image should not be deleted as they claimed that it was a breach… Read more »