Wikimedia engineering September 2011 report

Major news in September include:

Hover your mouse over the green question marks ([?]) to see the description of a particular project.

Upcoming events

  • New Orleans hackathon (14–16 October, New Orleans, USA) — Ryan Lane and Sumana Harihareswara continued to plan this coding event, which will focus on tools and infrastructure, and Wikimedia Labs. Discussions about the upcoming migration to git are also planned.
  • Weekend of Code (18 October – 8 November, online) — Weekend of code is a planned hacking contest targeting developers worldwide. Awards will be presented for the best gadget, extension, and mobile app. The contest will be advertised via different channels.
  • India hackathon (18–20 November, Mumbai, India) — A hackathon will be held concurrently with the WikiConference India 2011, and near its venue. The main focus will be on language, mobile and offline support for MediaWiki content. About 100–150 participants are expected (read more).
  • Brighton hackathon (19–20 November, Brighton, England) — Lewis Cawte is preparing a hackathon in Europe dedicated to general MediaWiki hacking.
  • Check out the Software deployments page on the wikitech wiki for up-to-date information on the upcoming deployments to Wikimedia sites.

Job openings

Are you looking to work for Wikimedia? We have a lot of hiring coming up, and we really love talking to active community members about these roles.
The following positions are open: Product Manager (Analytics), QA Lead, Director of Features Engineering, Systems Engineer (Data Analytics), Software Developer (Rich Text Editing, Features), Software Developer (Front-end) and Software Developer (Back-end), Product Manager (Mobile), Software Developer (Mobile), Product Manager.
The following requests for proposals are open: Networking Contractor Amsterdam, Internationalization and Localization Feature Development, Logging Analysis, Article Feedback Feature, Development and Operations Engineer, Systems & Operations Engineer, XML Dumps Help.

Operations

Site infrastructure

  • Tampa Data Center [?] — Our new data center contractor in Tampa, Chris Johnson, has finalized the installation of three new racks, which will be used for networking, application serving, caching and data storage. Two racks (72 servers) worth of application servers have also been installed. With each server having 12 CPU cores, this is almost doubling our existing application server capacity. To-date, we have deployed and added 58 (of the 72) new application servers to production capacity.
  • Virginia Data Center [?] — Ben Hartshorne has completed replicating External storage (article texts) data to our new data center in Ashburn, for disaster recovery and usage by eqiad application servers in the future. Preparations are also being made to deploy bits.wikimedia.org in eqiad using Varnish 3. Jeff Green has been building up the new EQIAD fundraising infrastructure and securing those boundary servers.
  • HTTPS — Protocol-relative URLs have been enabled on all sites by Roan Kattouw. SSL termination servers have arrived and are in the process of being installed. HTTPS should be fully enabled in early October.

Testing environment

Backups and data archives

  • Data Dumps [?] — The September English Wikipedia run completed in under 8 days. The dump of all text revisions is now available in a number of smaller files rather than several giant ones. The dump system has been converted to work with heterogeneous deployment and has undergone some testing with the upcoming MediaWiki 1.18 codebase.

Features Engineering

Editing tools

Participation and editor retention

  • Article feedback [?]Dario Taraborelli continued to analyze the data recorded. Howie Fung assessed proposals from vendors in response to the RfP to build an extended review system, that would notably integrate free-text comments to the Article feedback feature.
  • LiquidThreads 3.0 [?]Andrew Garrett worked on the integration between the LiquidThreads interface and the new back-end. What remains now is mostly to re-implement administrative actions, logging integration, new messages, and other ancillary parts of LiquidThreads on top of the new back-end. Once this is done, there will be a new, more stable version available for use on Wikimedia wikis.
  • MoodBar [?]Timo Tijhof fixed bugs, notably with Internet Explorer. Howie Fung analyzed the data recorded; the users’ most frequent complaints and questions concerned page creation, file upload, the text editor, and having their edits reverted.
  • Feedback Dashboard [?]Brandon Harris expanded the scope of this dashboard to support different sources, and updated the design. Roan Kattouw started to implement the feature, aided by Trevor Parscal for the HTML and CSS part.
  • Article creation and patrol — In response to a request by a majority of English Wikipedia community members to restrict new page creation to autoconfirmed users, discussions are underway to improve the article creation workflow and the user interface for new page patrolling as alternative strategies to cope with the new page creation backlog and reduce high-friction interactions with new users.

Multimedia Tools

  • UploadWizard [?]Ian Baker, Neil Kandalgaonkar and Jeroen De Dauw fixed a number of bugs, notably related to the Wiki Loves Monuments campaign and the deployment of protocol-relative URLs to Wikimedia Commons. Ian also researched solutions for multi-file selection and AJAX uploading. Jeroen and Neil worked on a feature to support custom licenses.

MediaWiki infrastructure

  • ResourceLoader [?]Roan Kattouw completed the back-end for .js/.css page search suggestions, overhauled the Gadgets API, reviewed code, and greatly improved caching. Timo Tijhof continued to work on the gadget manager and the AJAX gadget editor, which is nearing completion: displaying, modifying, saving of gadgets and autocompletion in all form fields is now implemented. Roan and Timo also prepared a plan for a prototype setup to test shared gadgets in a wiki farm environment.

Wikimedia Labs

Mobile

  • Mobile Research [?]Mani Pande and Parul Vora continued to work on the report on their field research in India and Brazil. They completed phone interviews in San Francisco, Chicago and Dallas (with partner firm Answerlab) and started to schedule in-person interviews.
  • MobileFrontend [?] — MobileFrontend was deployed in September and is now the default Wikipedia mobile experience. We’ve reached out to our various communities to create custom main pages, and are rolling out new ways of viewing Wikimedia projects on mobile. We also migrated our old WAP gateway traffic to MobileFrontend, so that we can serve our users from one place. Next we’ll be iterating on our roadmap to guide our work.

Special projects

Fundraising support

  • 2011 Fundraiser [?]Ryan Kaldari added increased logging for CentralNotice changes, including interfaces and filters to search and review them. Katie Horn abstracted and refactored the DonationInterface extension in preparation for supporting a new potential payment processor, GlobalCollect. The team also fixed a number of bugs and added new features (see the retrospective of sprints 5 & 6). Jeff Green continued general work to improve the resiliency and reliability of the fundraising architecture.

Offline

  • Kiwix UX initiative [?] — The Kiwix team built out a beta portal for feedback this month, where users can now easily submit their thoughts on using Kiwix. Next, we’ll be adding some in-app feedback to offer multiple ways of sending ideas and feedback.

Platform Engineering

MediaWiki Core

  • MediaWiki 1.18 [?] — Wikimedia engineers worked together relentlessly in September to ready MediaWiki 1.18 for deployment. They finalized the review of the code, and fixed all the issues they could find. The deployment to Wikimedia sites was split into several phases using the heterogeneous deployment system. Stages 1 and 2 were completed on select wikis without too much trouble. Deployment to all remaining Wikimedia sites is scheduled for October 4th, 2011.
  • Code review management [?] — Even though engineering and code review efforts were focused on MediaWiki 1.18 in September, the backlog of unreviewed commits in trunk still continued to decrease (see chart), which means we will be able to release MediaWiki 1.19 fairly rapidly, possibly as soon as December 2011. Work on MediaWiki 1.19 is scheduled to start as soon as MediaWiki 1.18 is officially released.
  • API maintenance [?]Sam Reed continued to maintain the API and worked on integrating the new Wikimedia dashboard with external data sources.
  • Shell requests [?]Sam Reed continued to go through the backlog of shell requests and to process them. Because of the number of new requests, the backlog has remained around 60 open requests.
  • Continuous integration [?]Chad Horohoe worked with Daniel Zahn to set up the dedicated server in our Virginia data center. Its configuration was automated with Puppet. TestSwarm remains to be pupettized. The server is expected to be put in production in early October.
  • Wikitext scripting [?]Tim Starling did some prototyping and testing on this project. He created a test harness that can be used to trial the various approaches, and made a Wikitext-to-Lua template converter to test the relative performance.
  • Git conversion [?]Rob Lanphier proposed a draft plan to proceed to a migration to git over the following months, and got a positive response from developers.

Wikimedia analytics

Technical Liaison; Developer Relations


This article was written collaboratively by Wikimedia engineers and managers. See the revision history and associated status pages. A wiki version is also available.

Archive notice: This is an archived post from blog.wikimedia.org, which operated under different editorial and content guidelines than Diff.

1 Comment
Inline Feedbacks
View all comments

[…] highlights from the Wikimedia Foundation Report and the Wikimedia engineering report for September 2011. Please consider helping with translations into German, French, Spanish, […]