Wikimedia Research Newsletter, October 2016

“Gender gap on Wikipedia: visible in all categories?”

Reviewed by Giuseppe Profiti

Asteroids are among the categories with the most overrepresentation of male editors, and figure skating among those with most female overrepresentation

This bachelor thesis^[1] looks for gender imbalance among editors for specific categories in the English Wikipedia. The analysis is based on the edits of users who publicly disclosed their gender (about 176 thousand) to more than 3.7 million articles in 470 categories (derived from DBpedia‘s ontology, rather than Wikipedia’s inbuilt category system). The thesis first establishes the distribution of editors by gender (roughly 85% males and 15% females). The number of edits by each group is statistically compared to that baseline distribution. For each category, if it varies from the baseline, it is considered to represent a gender gap, i.e. that editors from that gender are overrepresented in that category.

The results show that despite the huge imbalance in the two groups, pages in some categories receive more edits from users belonging to one gender, while other categories are dominated by the other one. As the “Top five categories where male editors are most overrepresented”, the author lists “YearInSpaceflight”, “Asteroid”, “BaseballSeason”, “MotorsportSeason”, and “FormulaOneTeam”. He observes sports as recurring theme “throughout all significant ‘male categories’. Besides sports other recurring subjects are transport and politics.” On the other hand, “the categories with a female overrepresentation show somewhat less obvious recurring themes. Many of these categories are more or less culture related however.” The five categories with the most female overrepresentation are “FigureSkater”, “Skater”, “Garden”, “GaelicGamesPlayer”, and “Mollusca”.

While highlighting some information on such unbalanced distribution, the underlying hypothesis could be further explored by using the quantity of text changed in each edit and other patterns mentioned by the author.

(See related Signpost coverage from 2011: “New tool analyzes article contributors’ gender and location“)

Quality and importance in different language editions

Reviewed by Morten Warncke-Wang

While much is known about the quality of Wikipedia articles, less is known about how the different language editions assess article importance. The English Wikipedia’s article about waffles is for instance labelled “top-importance” by WikiProject Breakfast, the highest category possible, but at the same time labelled “high importance” by WikiProject Food and Drink (you can find both of these labels on waffle’s talk page). A paper at the International Conference on Information and Software Technologies studies titled “Quality and Importance of Wikipedia Articles in Different Languages”^[2] studies the connection between importance and quality. The paper’s three research questions look at whether importance affects quality, what parameters are useful for applying machine learning to automatically assess importance, and if there are differences between how language editions model importance.

The English edition offers the most data on article importance, and the paper therefore uses a dataset of English articles to test if importance affects quality. Using a random forest classifier and a model with 85 parameters, a modest increase in classifier performance is found when importance is added as a parameter, indicating that importance affects quality. The same dataset and model is then trained to predict article importance, finding that about two-thirds of top- and low-importance articles can be correctly identified. Lastly the paper compares the importance of model features between different language editions, finding many differences, although these are not described in more detail.

Research on aspects of article quality across different language editions is an area that has not received a lot of attention, making this paper a welcome addition to the literature. It is also great to see article importance being studied. At the same time, this paper could have made a much stronger contribution through comparisons against a sensible baseline (this reviewer notes that the paper cites an in-press paper by the same authors^{[supp 1]}, although that paper’s results do not appear to be available in English) because the classifier performance appears to be similar to for instance ORES although ORES uses a model with a lot fewer parameters. A deeper investigation into article importance would also be worthwhile, for example because importance differs between topic areas, as exemplified by the article on waffle described earlier.

Why women edit less: a controlled experiment

Reviewed by Jonathan Morgan

Researchers have attempted to quantify Wikipedia’s gender gap and its impact on content type and quality, and to understand the reasons for the gender gap. A new journal article^[3] attempts to experimentally evaluate several hypotheses for why women tend to edit Wikipedia less than men do.

The researchers asked 192 male and female college students to contribute a draft essay about school bullying. The version of the draft that participants were asked to work on had already been edited by four other users (secretly, the researchers themselves), identified by pseudonyms. Two of the pseudonyms were obviously gendered (“Ms Trouble”, “Mr Football”), and two were gender neutral (“Cheerios4Life”, “AnonymousOne”). Since most people are not familiar with the mechanics of wiki editing, the researchers used a Microsoft Word document with “track changes” enabled as a platform for the editing task, to simulate the versioning and commenting capabilities of MediaWiki pages. The researchers also surveyed the students to gather relevant demographic and psychometric data, and compared their survey responses with their editing behaviors.

Findings from this study include that while women edited more than men overall (contributed more words to the draft), they were less likely to edit under the conditions designed to approximate the social environment of Wikipedia. Specifically, women edited less where there were few or no female-identified collaborators present, and where feedback from the pseudonymous collaborators was neutral (vs. constructive). Interestingly, female participants also tended to assume that one of the non-gendered pseudonyms (“AnonymousOne”) was male, and also evaluate feedback from that editor as more critical than male participants who received the same feedback. Based on these findings, the researchers suggest that increasing the visibility of female editors and encouraging constructive feedback may encourage more women to edit Wikipedia.

“Wikipedia traffic data and electoral prediction: towards theoretically informed models”

Reviewed by Zareen Farooqui

This research^[4] aims to explore the relationship between Wikipedia page view statistics and electoral results during the 2009 and 2014 European Parliament elections in regards to overall voter turnout and individual party results. The article suggests two reasons why voters might seek information: to research new parties which a voter beyond the voter’s familiarity, and to research alternative party options if a voter is unhappy with the party they previously supported (thus becoming swing voters).

The first dataset used in this research is Wikipedia page views data on the general election page in 14 different languages (those which are the primary languages of the voting countries). The second dataset includes political parties which had at least 5% vote share in the 2009 and/or 2014 elections in the UK, France, Germany, Spain and Italy. The researchers gathered additional data points such as number of views to the political party’s Wikipedia page the week before the election, the final percentage of vote share each party received, whether a party was new, whether a party was incumbent, and the number of times each party was mentioned in print media during the week before the election.

Comparing the relative change in page views to the EU Parliament elections article and total voter turnout in the 2009 and 2014 elections indicates that interest in election events is proportional to levels of readership on Wikipedia. This research suggests that often the party garnering the most page views does not win the election, rather, it may be a smaller party which interested swing voters. Figure 1(a) shows a high correlation between print media mentions and overall voter share for parties. Figure 1(b) shows Wikipedia page views may predict a new party’s success, while news outlet mentions are better at predicting an established party’s success.

The research tests the theory that an increase in Wikipedia page views may suggest an increase to votes for a party using three linear ordinary least squares regression models. The first model is a baseline of past voting results. The second model is also a baseline model which includes past voting results, along with all other non-Wikipedia related data collected. These baseline models serve as a comparison to the third model, which includes all the previously modeled data, along with two Wikipedia-related parameters. The models show that Wikipedia can be considered a predictor of voter outcome, but it only marginally improves upon the baseline models. Wikipedia’s predictive power lies in predicting the amount a party’s vote share may increase or decrease from the previous election cycle.

As noted by the researchers, one limitation of this article is that the data is at an aggregated level, while all theories are at the micro level. Also, it is unclear what number of Wikipedia page views reflect voters versus other groups, such as journalists or those those affiliated with the parties.

(See also our 2014 coverage of some related blog posts by the same authors: “Wikipedia use driven by news media or replacing news media?“)

Briefly

Conferences and events

See the research events page on Meta-wiki for upcoming conferences and events, including submission deadlines.

Other recent publications

Other recent publications that could not be covered in time for this issue include the items listed below. contributions are always welcome for reviewing or summarizing newly published research.

• “Disinformation on the Web: impact, characteristics, and detection of Wikipedia hoaxes”^[5] From the abstract: “We find that, while most hoaxes are detected quickly and have little impact on Wikipedia, a small number of hoaxes survive long and are well cited across the Web. Second, we characterize the nature of successful hoaxes by comparing them to legitimate articles and to failed hoaxes that were discovered shortly after being created. We find characteristic differences in terms of article structure and content, embeddedness into the rest of Wikipedia, and features of the editor who created the hoax. Third, we successfully apply our findings to address a series of classification tasks, most notably to determine whether a given article is a hoax. And finally, we describe and evaluate a task involving humans distinguishing hoaxes from non-hoaxes. We find that humans are not good at solving this task and that our automated classifier outperforms them by a big margin.”
• “Where are the women in Wikipedia? Understanding the different psychological experiences of men and women in Wikipedia”^[6] From the abstract: “We analyzed data from a sample of 1,598 individuals in the United States who completed the English version of an international survey of Wikipedia users and readers conducted in 2008 and who reported being occasional contributors. … Women reported less confidence in their expertise, expressed greater discomfort with editing (which typically involves conflict) and reported more negative responses to critical feedback compared to men. Mediation analyses revealed that confidence in expertise and discomfort with editing partially mediated the gender difference in number of articles edited, the standard measure for contribution to Wikipedia.” (See also our 2012 coverage of a related paper by the same authors: “Gender gap connected to conflict aversion and lower confidence among women“)
• “Wikipedia and stock return: Wikipedia usage pattern helps to predict the individual stock movement”^[7] From the abstract: “We provide evidence that data on how often a company’s Wikipedia page is being viewed is linked to its subsequent performance in the stock market. We then develop a portfolio in line with the Wikipedia usages and demonstrate that our investment strategy based on Wikipedia views is profitable both financially and statistically.”
• “Editing diversity in: reading diversity discourses on Wikipedia”^[8] From the abstract: “… the Wikimedia Foundation (WMF) has devoted a fair amount of time and resources to tackling [Wikipedia’s] ‘gender gap.’ While we acknowledge the good intentions of the WMF and volunteer efforts to improve conditions for women editors on Wikipedia, we argue that borrowing from corporatized diversity initiatives more effectively supports organizational growth rather than addresses the underlying reasons behind women’s low representation and participation.”
• “Circadian patterns on Wikipedia edits”^[9] From the abstract: “We … show in this work that Wikipedia editing presents well defined periodic patterns with respect to daily, weekly and monthly activity. In addition, we also show the periodic nature of the number of inter-event in time.”
  From the rest of the paper: “Our data sample is a database of WP edits, of pages written in English in the period of about 10 years ending in January 2010 … In general, [the 100 most active] editors have the main power peak at ∼ 1.157 × 10⁻⁵ Hz corresponding to a period of 24 h and a second peak at ∼ 2.315 × 10⁻⁵ Hz, matching a 12 h period, a harmonic from the main frequency. … The highest activity peak can switch between mornings and evenings, depending on the day. In the process of WP editing, the change of activity patterns on week-ends is clear. … Along the year the intensity of activity seems conditioned by holidays.”
  See also our 2011 coverage of a related paper: “Wikipedians’ weekends in international comparison”
  

  Slide from the August 2016 Wikimedia Research showcase presentation about the fact checking research: The shortest path from the article Barack Obama (left) to socialism (right) passes through some nodes with high degree that represent generic entities, indicating that statements such as “Barack Obama is a socialist” have low truth value.

• “Computational fact checking from knowledge networks”^[10] From the abstract: “we show that the complexities of human fact checking can be approximated quite well by finding the shortest path between concept nodes under properly defined semantic proximity metrics on knowledge graphs. Framed as a network problem this approach is feasible with efficient computational techniques. We evaluate this approach by examining tens of thousands of claims related to history, entertainment, geography, and biographical information using a public knowledge graph extracted from Wikipedia. Statements independently known to be true consistently receive higher support via our method than do false ones.”
• “Challenges of mathematical information retrieval in the NTCIR-11 Math Wikipedia Task”^[11] From the abstract: “… the optional Wikipedia Task provides a test collection for retrieval of individual mathematical formula from Wikipedia based on search topics that contain exactly one formula pattern. We developed a framework for automatic query generation and immediate evaluation.”
• “Quantifying the relationship between hit count estimates and Wikipedia article traffic”^[12] From the abstract: “This paper analyzes the relationship between search engine hit counts and Wikipedia article views by evaluating the cross correlation between them. We observe the hit count estimates of three popular search engines over a month and compare them with the Wikipedia page views. The strongest cross correlations are recorded with their delays in days.”
• “LeadWise: using online bots to recruite and guide expert volunteers”^[13] From the abstract: “we propose LeadWise, a system that uses social media bots to recruit and guide contributions from experts to assist non-profits in reaching their goals. … We focus in particular on experts who can help Wikipedia in its objective of reducing the gender gap by covering more women in its articles. Results from our first pilot show that LeadWise was able to obtain a noteworthy number of expert participants in a two week period with limited requests to targeted specialists.”
  From the rest of the article: “We created ‘CauseBots’ [on Twitter] which are bots that present themselves as a social cause (hiding that the accounts are an automated agent). We also created ‘AgentBots’ which are bots that present themselves as bots supporting a social cause. … the first thing all of LeadWise’s bots do is build a ‘supportive audience’ with experts. … Once [they have] a supportive audience with over fifteen members, the bots follow the same behavioural rules to request and guide participation: They publicly ask for the names of women who should be added to Wikipedia. … We primarily focused on Spanish speaking experts in gender equality. … We considered that experts were individuals who tweeted heavily about gender equality. Both bots looked for users mentioning related Spanish keywords, such as ‘equidad de genero,’ and who had already published a large number of related tweets (over 50). … In total, 22 new women were added [by these experts recruited on Twitter] to the list of Wikipedia articles to cover.” TB

References

Supplementary references and notes: