“Rate this Page” is Coming to the English Wikipedia

Translate This Post

Since May, the Article Feedback Tool has been available on 100,000 English Wikipedia articles (see blog post). We have now kicked off full deployment to the English Wikipedia at a rate of about 370,000 articles per day and will continue at this rate until deployment is complete.

Aft_phase_2
Ratings interface for the Article Feedback Tool

We wanted to take a moment to briefly recap what we’ve learned so far, what lies ahead, and how we can work with the community improve this feature.   Features like Article Feedback can always be improved, so we will continue to experiment, measure, and iterate based on user and community feedback, testing, and analysis of how the feature is being used.
Rating data from the tool is available for your analysis — please dig in and let us know what you find. Toolserver developers can also access the rating data (minus personal information) in real-time to develop new dashboards and views of the data.

What We’ve Learned So Far

 
Readers like to provide feedback. The survey we’re currently running shows that over 90% of users find the ratings useful.  Many of these raters see the tool as a way to participate in article development — when asked why they rated and article, over half reported wanting to “positively affect the development of the page.”
 
AFT_survey_combined(1)
Users of the feedback tool also left some enthusiastic comments (as well as some critical ones) about the tool. For example:

The option to rate a page should be available on every page, all the time, once per page per user per day.

As a high school librarian, I want my students to assess the sources of information they use.  This feature forces them to consider the reliability of Wiki articles.  Glad you have it.

Ratings seem like an interesting idea, I feel like the metrics used to determine the overall value of the page are viable, and I’ll be interested to see how the feature fares when it’s rolled out and has some miles under its belt.

The vast majority of raters were previously only readers of Wikipedia.  Of the registered users that rated an article, 66% had no prior editing activity.  For these registered users, rating an article represents their first participatory activity on Wikipedia.  These initial results show that we are starting to engage these users beyond just passive reading, and they seem to like it.
The feature brings in editors. One of the main Strategic Goals for the upcoming year is to increase the number of active editors contributing to WMF projects.  The initial data from the Article Feedback tool suggests that reader feedback could become a meaningful point of entry for future editors.
Once users have successfully submitted a rating, a randomly selected subset of them are shown an invitation to edit the page. Of the users that were invited to edit, 17% attempted to edit the page.  15% of those ended up successfully completing an edit.  These results strongly suggest that a feedback tool could successfully convert passive readers into active contributors of Wikipedia.  A rich text editor could make this path to editing even more promising.
While these initial results are certainly encouraging, we need to assess whether these editors are, in fact, improving Wikipedia.  We need to measure their level of activity, the quality of their contributions, their longevity, and other characteristics.
Ratings are a useful measure of some dimensions of quality.  In its current form, the Article Feedback Tool appears to provide useful feedback on some dimensions of quality, while the usefulness of the feedback on other dimensions of quality is still an open research question. Completeness and Trustworthy (formerly “Well-Sourced”) appear to be dimensions where readers can provide a reasonable level of assessment.  Research shows that ratings along these dimensions are correlated with the length and amount of citations, respectively.  We need to determine whether the ratings in “Objective” and “Well-Written” meaningfully predict quality in those categories. We released public dumps of AFT data and would love to hear about new approaches of measuring how well ratings reflect article quality.
We received feedback from community members on how to improve the feature. We’ve received a fair amount of feedback from the community on the usefulness of AFT, mainly through IRC Office Hours and on the AFT discussion page.  There have been many suggestions on how to make the feedback tool more valuable for the community.  For example, the idea of having a “Suggestions for Improvement”-type comment box has been raised several times.  Such a box would enable readers to provide concrete feedback directly to the editing community on how to improve an article.  We plan to develop some kind of commenting system in the near future.

aft-extra(1)
Illustration of a potential "Suggest Improvements" feature

AFT could help surface problematic articles in real time, as well as articles that may qualify for increased visibility. We’ve started experimenting with a dashboard for surfacing both highly rated and lowly rated articles.   Ultimately, the dashboard could help identify articles that need attention (e.g., articles that have been recently vandalized) as well as articles that might be considered for increased visibility (e.g., candidates for Featured Articles).  We will continue to experiment with algorithms that help surface trends in articles that may be useful for the editing community.

Next Steps

Over the coming weeks, we will continue to roll out the Article Feedback Tool on the English Wikipedia.  Once this rollout is complete, we will start planning the next version of the tool.  For those interested in following the discussion, we will be documenting progress on the Article Feedback Project Page.  We would love to get your feedback (pun intended!) on how the feature is being used, what’s working, and what might be changed.  We also encourage folks to dig into the data.  Once the feature is fully deployed, there will be mountains of data to sift through and analyze, which will be a boon to researchers and developers alike.
We’d especially like to encourage members of the community to get involved in the further development of the feature.  If you’re interested in getting involved (e.g., design input, data analysis/interpretation, bug-squashing, etc.), please drop a note on the project talk page.

Howie Fung, Senior Product Manager
Dario Taraborelli, Senior Research Analyst
Erik Moeller, VP Engineering and Product Development

Archive notice: This is an archived post from blog.wikimedia.org, which operated under different editorial and content guidelines than Diff.

Can you help us translate this article?

In order for this article to reach as many people as possible we would like your help. Can you translate this article to get the message out?

22 Comments
Inline Feedbacks
View all comments

Does this have a facility to add questions about the user: Age? Education? Income? Continent/Region? Employment? Sex? It could be useful to apply some demographics to the quality judgements – Do female university students from Latin America rate articles differently from male pensioners from south east asia

I’ve been calling for this reform for years. And Wikipedis is indeed slow to innovate. It isn’t social media. It has had no “likes” or open public forums or voting system. The notion that “anyone” can edit a page is bogus as anyone who tries to do so finds out, as there is a very arcane set of rules for editing and a cabal at the center of the cult that makes the judgement calls on most of the controversial entries. If an encyclopedia company did that with an editorial board, we would dub it old-fashioned and backward. Wikipedia isn’t… Read more »

[…] Love it or hate it, you can't say Wikipedia is slow to innovate. The giant encyclopedia site announced this weekend that it will now roll-out site-wide an article rating system that allows page visitors to rate an […]

Yes. This new feature will really improve the quality of the content based on users recommendation.

[…] new Wikipedia ranking system allows users to rate the reliability of individual articles.Wikipedia announced this weekend plans to launch a new rating system site-wide that will allow readers to rank the […]

[…] announced this weekend plans to launch a new rating system site-wide that will allow readers to rank the […]

[…] artykułów. Ponieważ informacja zwrotna co do systemu oceniania była bardzo pozytywna, Wikimedia ogłosiła wprowadzenie go dla wszystkich anglojęzycznych artykułów w tempie 370 000 dziennie. Z […]

[…] Wikipédia anunciou recentemente que vai disponibilizar em todo o site um sistema de rating que permitirá que os visitantes […]

[…] de prueba en mayo y ahora lo implanta en la edición en inglés. Wikipedia ha introducido un sistema de estrellas que permite al lector puntuar los contenidos. Lo probó en 100.000 entradas y ahora anuncia que lo […]

[…] de prueba en mayo y ahora lo implanta en la edición en inglés. Wikipedia ha introducido un sistema de estrellas que permite al lector puntuar los contenidos. Lo probó en 100.000 entradas y ahora anuncia que lo […]

[…] de prueba en mayo y ahora lo implanta en la edición en inglés. Wikipedia ha introducido un sistema de estrellas que permite al lector puntuar los contenidos. Lo probó en 100.000 entradas y ahora anuncia que lo […]

[…] de prueba en mayo y ahora lo implanta en la edición en inglés. Wikipedia ha introducido un sistema de estrellas que permite al lector puntuar los contenidos. Lo probó en 100.000 entradas y ahora anuncia que lo […]

[…] This weekend, the Wikimedia Foundation started adding it to about 370,000 articles per day, and will continue until it’s available with all of the almost 3.7 million English-language articles on Wikipedia, the group said in a blog post. […]

[…] Wikipedia found that most people who rated pages had never before edited anything. So by adding this feedback tool, they could make editors of readers. Although only a small subset made an edit—15 percent of those who were “invited” to edit a page after submitting a rating successfully did so—given the volume of Wikipedia’s readers that adds up. […]

[…] Wikipedia found that most people who rated pages had never before edited anything. So by adding this feedback tool, they could make editors of readers. Although only a small subset made an edit – 15 per cent of those who were “invited” to edit a page after submitting a rating successfully did so – given the volume of Wikipedia’s readers that adds up. […]

[…] Wikipedia found that most people who rated pages had never before edited anything. So by adding this feedback tool, they could make editors of readers. Although only a small subset made an edit—15 percent of those who were “invited” to edit a page after submitting a rating successfully did so—given the volume of Wikipedia’s readers that adds up. […]

[…] de prueba en mayo y ahora lo implanta en la edición en inglés. Wikipedia ha introducido un sistema de estrellas que permite al lector puntuar los contenidos. Lo probó en 100.000 entradas y ahora anuncia que lo […]

[…] Wikipedia anunciou em seu blog oficial que vai adotar o sistema “Rate this Page” em seus artigos. A enciclopédia mais famosa […]

[…] Wikipedia found that the majority those who rated pages had never before edited anything . So by adding this feedback tool, they may make editors of readers. Although just a small subset made an edit-15 percent of these who were “invited” to edit a page after submitting a rating successfully did so-given the amount of Wikipedia’s readers that adds up. […]

[…] can read the official ‘Article Feedback Tool’ announcement on Wikipedia (Wikimedia) blog. Tweet (function() { var s = document.createElement('SCRIPT'), s1 = […]

[…] Wikipedia quietly launched a new feature to its over 3.7 million English-language entries: the article feedback tool. An inconspicuous box at the bottom of articles, it asks you to “rate this page” on a scale of […]

So 15 % of those invited edited. What percentage of those ”not” invited edited? And is any analysis of the quality and long term contribution patterns planned?