Seven years after Nature, pilot study compares Wikipedia favorably to other encyclopedias in three languages

Improving the quality of articles has long been one of the primary aims of contributors to Wikipedia, and is one of the Wikimedia movement’s 2010-15 strategic priorities, but measuring it objectively has remained a challenge. In 2005, Nature famously reported that Wikipedia articles on scientific topics contained just four errors per article on average, compared to three errors per article in the online edition of Encyclopaedia Britannica. Britannica objected to the report, but Nature stood by it, and the report remains widely cited today.

Since that time, however, there have been relatively few independent analyses of Wikipedia article quality, despite the enormous growth of the project. Wikipedia today counts more than 23 million articles across languages (more than 4 million articles in the English Wikipedia alone) compared to 3.7 million total articles in 2005; today it ranks 6th by overall traffic according to Alexa, while it ranked 37th in 2005.
With increase in size and reach, how has quality evolved? How does Wikipedia compare today to other online encyclopedias, quality-wise? And what are good methods to measure the quality of encyclopedic articles?
The Wikimedia Foundation is announcing the release of a pilot study conducted by Epic, an e-learning consultancy, in partnership with Oxford University – “Assessing the Accuracy and Quality of Wikipedia Entries Compared to Popular Online Alternative Encyclopaedias: A Preliminary Comparative Study Across Disciplines in English, Spanish and Arabic.”
The study compared a sample of English Wikipedia articles to equivalent articles in Encyclopaedia Britannica, Spanish Wikipedia to Enciclonet, and Arabic Wikipedia to Mawsoah and Arab Encyclopaedia. 22 articles in the sample were blind-assessed by 2 to 3 native speaking academic experts each, both quantitatively and qualitatively.
The small size of the sample does not allow us to generalize the results to Wikipedia as a whole. However, as a pilot primarily focused on methodology, the study offers new insights into the design of a protocol for expert assessment of encyclopedic contents. For our editor community and for the Foundation, which commissioned the study in 2011, it also offers evidence to inform the design of quality assessment mechanisms and quality metrics that may be used on Wikipedia itself.
The results suggest that Wikipedia articles in this sample scored higher altogether in each of the three languages, and fared particularly well in categories of accuracy and references. As the report notes, the English Wikipedia fared well in this sample against Encyclopaedia Britannica in terms of accuracy, references and overall judgement, with little differences between the two on style and overall quality score. Similar results were found when comparing Wikipedia articles in Spanish to Enciclonet. In Arabic, Mawsoah and Arab Encyclopaedia articles scored higher on style than Wikipedia, but no significant differences were found on accuracy, references, overall judgment and overall quality score. None of the encyclopedias considered in this study were rated highly by the academics in terms of suitability for citation in academic publications.
We hope that the results of this study will encourage further independent research on the quality of Wikipedia articles. To this end, Epic and Oxford University are releasing the full version of the report of this study under a Creative Commons Attribution-Share Alike license. They have announced the report here and have released an anonymized dataset under a Creative Commons Zero dedication. The team welcomes comments and feedback on the talk page of the project.
We are very encouraged by the results for this small sample of Wikipedia articles in three languages. While pointing the way forward for further research, these results affirm the quality of the collaborative work of our editor community.

 
Dario Taraborelli, Senior Research Analyst

Archive notice: This is an archived post from blog.wikimedia.org, which operated under different editorial and content guidelines than Diff.

26 Comments
Inline Feedbacks
View all comments

This study ignores two significant deficiencies of Wikipedia as it compares with references like Encyclopedia Britannica. First, coverage of topics within a broad subject area is determined by self-selecting submitters, so there are often coverage gaps. Second, there are often inconsistencies from one article to the next within a broad subject area. Although individual articles may be accurate with respect to their topic, one cannot rely on the collection of articles for a broad subject area as consistent or comprehensive. An actively edited reference like EB is much less likely to be deficient this way.
More about this at
http://www.robertvbinder.com/britannica-brat/

Thank you for the link. Opinions of the form “Encyclopedia X is produced by method Y, therefore its content must exhibit property Z” are very frequent in discussions comparing Wikipedia with other encyclopedias. I think one of the values of the present study is that it adheres to the established principle of blind review, and developed methods (described in section 3.4.2) to ensure that reviewers’ judgments are not biased by such preconceived opinions about each encyclopedia’s production process. I agree that besides factual accuracy, comprehensiveness/evenness of coverage (“not what was there, but what was not”, as you wrote in your… Read more »

Bob “First, coverage of topics within a broad subject area is determined by self-selecting submitters, so there are often coverage gaps” Citation needed Bob. My impression is that coverage gaps on English Wikipedia have been filled over time. “Second, there are often inconsistencies from one article to the next within a broad subject area. Although individual articles may be accurate with respect to their topic, one cannot rely on the collection of articles for a broad subject area as consistent or comprehensive.” Citation needed again Bob. Do you have evidence for this or is it just your impression? My impression… Read more »

Thanks for this excellent news.
I have begun transcribing this freely licensed publication on Wikisource:
http://en.wikisource.org/wiki/Index:EPIC_Oxford_report.pdf
…and would welcome any help! The goal is to make a more accessible HTML version of the study, to supplement the PDF. If anybody would like some pointers on how to get started with this, contact me at http://en.wikisource.org/wiki/User_talk:Peteforsyth

And, I’m finding errors — nice case in point why using a wiki for things like this can be beneficial. (The first entry in the Table of Contents has the wrong page number; and the first citation in the paper appears to be absent.)

[…] un desempeño especialmente bueno en cuanto a exactitud y provisión de referencias”, escribe Dario Taraborelli, investigador de la Wikimedia Foundation. Según el informe, la Wikipedia en inglés se compara […]

[…] un desempeño especialmente bueno en cuanto a exactitud y provisión de referencias”, escribe Dario Taraborelli, investigador de la Wikimedia Foundation. Según el informe, la Wikipedia en inglés se compara […]

[…] al report pubblicato da EPIC, le voci enciclopediche di Wikipedia sarebbero più aggiornate delle altre, con […]

[…] While there are often jokes about Wikipedia’s accuracy, a study (albeit, commissioned by Wikimedia) came out last week finding that the site compares favorably to other encyclopedias. […]

“Enciclonet was selected because of its high popularity, its high Alexa traffic rank of 322,628” – That number is high indeed, but in a bad sense.

I’ve read some fifty pages of the study. One of the alleged weaknesses of Wikipedia, lack of definition of terms, has an easy explanation: Wikipedia relies on users clicking links to get more information. For example, the article Ecology doesn’t really need to define organism, environment, biomass, ecosystem, species, community and biodiversity, because their definitions are a click away. The study treats articles as standalone works, but Wikipedia doesn’t work that way.

[…] Read more about the findings of the pilot study here and to access the full report, click here. […]

[…] un desempeño especialmente bueno en cuanto a exactitud y provisión de referencias”, escribe Dario Taraborelli, investigador de la Wikimedia […]

Bob, I can see how you might think that about Wikipedia, but I don’t think what you’re seeing are actually coverage *gaps* but rather the *much deeper coverage* that Wikipedia gives to certain topics than Britannica. The biggest difference between the Wikipedias and encyclopedias like Britannica — which didn’t show up in this study because it was restricted to areas where “articles from different online encyclopaedias were of comparable substance and focus” — is they are much, much, much larger than things like Britannica. Wikipedias are much, much larger, and much more comprehensive. As your article evocatively points out, Britannica… Read more »

This report, and in general all reporting on these comparisons, suffer in my view from having ignored the obvious discrepancies in validity expected from a study based on sampling, versus ones designed based on the actual manner in which Wikipedia is used. Any sample-basd approach, random or otherwise, ignores the fact that users do not come to an Encyclopedia and evaluate a random selection of content, rather, they come for information on one or a few subjects, often times embedded within larger articles, and so require uniform source quality and uniform “intra-source” navigation; they come for expertise on a very… Read more »

[…]  Britannica vs Wikipedia 2012 […]

[…] So lets begin with a definition “Politics (from Greek: πολιτικός politikos, definition “of, for, or relating to citizens”) is the making of a common decision for a group of people, that is, a uniform decision applying in the same way to all members of the group. It also involves the use of power by one person to affect the behavior of another person. More narrowly, it refers to achieving and exercising positions of governance— organized control over a human community, particularly a state. Furthermore, politics is the study or practice of the distribution of power and resources within a… Read more »

[…] Seven years after Nature, pilot study compares Wikipedia favorably to other encyclopedias in three l… […]

[…] These methods are so successful that they enable a crowdsourced website like Wikipedia to have essentially the same quality as the Encyclopedia Britannica. If you go to the open source coding site GitHub, where the Data […]

It’s actually very difficult in this busy life to listen news on Television,
so I only use web for that purpose, and take the latest information.

[…] for not being accurate or bad for “being editable by anyone” and not only experts, many studies have found it to be on par with other encyclopedias regarding accuracy, bias, and error […]

[…] Dario Taraborelli, “Seven years after Nature, pilot study compares Wikipedia favorably to other encyclopedias in three languages”, Wikimedia Foundation, web. Published: 2012.08.02; consulted: 2016.08.09. URL: https://diff.wikimedia.org/2012/08/02/seven-years-after-nature-pilot-study-compares-wikipedia-favora…. […]

[…] a blind study comparing the quality of content found on Wikipedia and Encyclopædia Britannica. The study found that there was very little difference in the quality of the content. Wikipedia scored even […]

[…] a blind study comparing the quality of content found on Wikipedia and Encyclopædia Britannica. The study found that there was very little difference in the quality of the content. Wikipedia scored even […]