Since December 2011, the Wikimedia Foundation has been testing a new version of the Article Feedback Tool, a feature first introduced on the English Wikipedia in 2010. The goal of version 5 (AFTv5) is to engage Wikipedia readers to become more active contributors, by inviting them to provide feedback on articles they read, and encouraging them to become editors over time.
Early tests of AFTv5 helped us answer the question of what design of the tool produces a desirable balance between volume and usefulness of the feedback collected. In this post we report results from two additional experiments designed to answer the following questions:
- Does a prominent invitation to use the tool affect the usefulness of submitted feedback?
- How does an invitation to leave feedback affect the conversion of readers into editors?
Our findings suggest that a prominent invitation to post feedback converts a significant number of readers into editors. These new editors appear less productive than other first-time Wikipedians; but their feedback appears just as useful, as below. These findings suggest that article feedback can increase the number of new editors on Wikipedia and can also help existing editors improve the encyclopedia based on reader feedback.
Prominence of Feedback Invitation
Like AFTv4, the AFTv5 form was originally positioned beneath the article content. This design intentionally requires that a reader (and potential feedback submitter) scroll past the entire article before submitting feedback about it. While this placement may act as a natural quality filter, many readers with relevant feedback may not find the article feedback form so far down on the page.
Increasing the visibility of the feedback form could be an effective way of increasing the rate at which feedback is submitted, although it might also elicit more irrelevant or lower quality feedback.
To explore this potential trade-off between quality and quantity of feedback submissions, we randomly split readers into three experimental groups: one group (control 1X) was presented just with the default AFTv5 form at the bottom of the article; the two other groups were shown additional shortcut links (1A and 1E) to the feedback form that allow a reader to submit feedback without scrolling to the bottom of the page.
Volume and quality of feedback with/without prominent links: The effect of additional, prominent links to the feedback form on the total volume of feedback collected (left) and on the proportion of useful feedback (right) (data collected between April 5-19, 2012).
A group of experienced Wikipedians judged the usefulness of this feedback for editors to improve Wikipedia articles via a blind assessment tool. The results were surprising: while the most prominent link condition (1E) elicited nearly twice as much feedback compared to the control group (191.4%), we found that the proportion of useful feedback remained about the same, when we had expected a significant decrease in quality.
Conversion
A potential risk in displaying an invitation to leave feedback (or any non-edit-based mode of participation) is that a reader’s inclination to edit the article directly could be cannibalized by the invitation to submit feedback. Soliciting feedback, one may argue, will violate the community’s “SOFIXIT” principle that encourages anyone to directly edit and improve an article. Given this concern, we might expect fewer potential new editors to make the leap from passive reader to active contributor by making their first edit to an article when AFT was available.
To look for such an effect and explore a potential stop-gap measure in the form of an invitation to edit, we performed a second experiment. We randomly assigned readers to three experimental groups. The control group (0X) saw no feedback form at all. A second group (1E) saw the prominent link E as a shortcut to the feedback form at the bottom of the article; after posting feedback, users in this group were presented with a call to action to edit the article. The last group (4E) was presented with a direct call to action to edit the article, linked via a prominent shortcut E and replacing the feedback form.
Conversion rate and productivity of new editors. The raw number of new users making their first edit by experimental condition (left) and the proportion of these editors making at least one productive edit in their first week (right). (data collected between April 27-May 7, 2012).
We narrowed the analysis to those editors who–to the best of our knowledge–made their first edit while participating in this experiment. We found that the number of users who completed their first edit via the standard “Edit links” (tab and section level) was consistent across the three experimental groups. Those users who followed the invitation to edit (whether direct or indirect) appear to supplemented the number of already converting readers.
We then measured the productivity of these new editors. We considered a newcomer productive if they made at least one edit that did not get reverted in their first week. We found that users who started editing via the edit links had the same productivity across the three experimental groups (~68%). However, those who started editing after viewing the invitation both directly (4E) and after submitting feedback (1E), were significantly less likely to be productive (~40%).
These results suggest that:
- The presence of a feedback form highlighted by a prominent link does not appear to cannibalize the conversion rate of readers who become new editors.
- The presence of an invitation to edit appears to elicit participation from new editors who are less likely to contribute productively, at least in the beginning of their wiki-career. But overall it still increased the absolute number of productive newcomers considerably.
A decision about whether such an approach is desirable should weigh the value of a larger number of productive new editors and useful feedback with the cost of an increase in unproductive edits.
What’s next
Wikimedia’s editor engagement team is preparing to deploy Article Feedback version 5 to about 10% of articles on the English Wikipedia in July 2012, for testing with a wider community. After final testing and code re-factoring, we plan to extend this new version to 100% of the English Wikipedia in early fall — then to other projects in the following months.
Based on these test results, we now project over 2 million feedback posts per month on the English Wikipedia when the tool is widely deployed later this year (on par with the total number of edits per month). In the process, article feedback can provide a whole new form of contribution for Wikipedia readers, most of whom do not currently edit the encyclopedia.
Note that in the process, we will retire the earlier Article Feedback version 4, which was based solely on user ratings, without comments — it was not as effective as this new version, according to our research.
You are welcome to try out Article Feedback v5 now, on one of these pages:
To learn more, read our help page. We welcome your comments, questions and suggestions in the comments below — or on the Article Feedback Talk page.
We hope you find this new tool helpful and look forward to hearing from you.
Dario Taraborelli, Sr Research Analyst, Strategy
Aaron Halfaker, Research Analyst (contractor)
Oliver Keyes, Community Liaison, Product Development (Contractor)
with
Fabrice Florin, Product Manager, Editor Engagement
Howie Fung, Director of Product Development
Wikimedia Foundation’s Editor Engagement Team
Can you help us translate this article?
In order for this article to reach as many people as possible we would like your help. Can you translate this article to get the message out?
Start translation