Wikimedia engineering December 2011 report

Translate This Post

Major news in December include:

Hover your mouse over the green question marks ([?]) to see the description of a particular project.

Events

Recent events

Upcoming events

  • Pune hackathon (10–12 February 2012, Pune, India) — Preparation began and registration opened for an outreach-focused developers week-end to take place in Pune, India, and led by Alolita Sharma. Approximately 70 participants are expected, focusing on the gadgets framework, mobile Wikimedia access, and internationalization.

Personnel

Job openings

Are you looking to work for Wikimedia? We have a lot of hiring coming up, and we really love talking to active community members about these roles.

  • Requests for proposals:
    • Executive Dashboard – Analytics — Help us improve and centralize the dashboard summarizing the most important data about the Wikimedia movement to understand overall community health.
    • XML Dumps — Help us improve the infrastructure used to build XML dumps of Wikipedia content, for backups and reuse by third parties.
    • Mobile UX — Help us redesign our mobile platform and apps as more and more visitors access Wikipedia and its sister sites via mobile devices.

Short news

  • Yuvaraj Pandian and Max Semenik joined the mobile team as contract developers.
  • Sara Smollett joined the operations team as a part-time contractor.
  • Diederik van Liere, formerly with the Community Department, is now helping the engineering department as a contractor for analytics work.

Operations

Site infrastructure

  • Data Centers [?] — The team deployed a new MediaWiki profiling system based on graphite, to track performance across the application stack, and to provide statistics/graphing as a service for MediaWiki within the WMF production environment. Some database servers were moved to newer hardware (including OTRS), and those in the Ashburn data center were upgraded to a new build of mysql-at-facebook. Mark Bergsma refactored our configuration tool (Puppet) to address scalability and performance issues.
  • Media Storage [?] — As part of the preparation for the migration of our media service to Swift, a distributed storage back-end, we need to keep the current system afloat a bit longer. We reclaimed some space by purging thumbnails not newly generated and not in use on any of our projects. We also performed Swift thumbnail integration and stress testing. Read performance is about 10x what we need on the performance test cluster so we’re good on that front. Write performance is only 2x what we need, but sufficient to move forward. Tests and research indicates performance drops over a few million objects; the easiest path forward is to shard the Commons container using the existing hashed characters in the URL, splitting the container into 256 containers.
  • HTTPS — HTTPS support was added for mobile, for Wikipedia. After an initial testing period, we’ll enable this for further mobile sites. A number of other miscellaneous services also had HTTPS set up or fixed.

Testing environment

  • Wikimedia Labs [?] — A server admin log was created for every project, as well as a combined log. OpenStackManager 1.3 and LdapAuthentication 2.0a were deployed to Labs. Live migration of instances has been enabled for the OpenStack Nova infrastructure, allowing updates and upgrade of hardware without bringing instances down. A gluster storage cluster has been ordered for use as volume storage. A number of projects were added or moved to Labs, including adminbot, nagios, Cluebot, testswarm and the reportcard service. There are now 33 projects, 52 instances, and 74 users.

Backups and data archives

  • Data Dumps [?] — The end of the year closed out with another full dump of the English language Wikipedia on schedule. More work was done on code to allow restart of the history phase of a dump from a specified point without a long catchup delay. An experimental service was tested this month: a newly formatted file of article content and an accompanying index, more convenient for data analysts and for use with offline readers.

Features Engineering

Editing tools

Participation and editor retention

Multimedia Tools

  • UploadWizard [?] — Users can now choose a default license for all uploads in their user preferences under “Upload Wizard” (bug 24702). All license choices now also link to the legal code of a license. The built-in feedback form more prominently links to Bugzilla.
  • TimedMediaHandler [?]Ian Baker and Neil Kandalgaonkar completed the review of all the code, including the transcoding part. They started to plan a test plan and a deployment to Wikimedia Labs.

MediaWiki infrastructure

  • ResourceLoader [?]Roan Kattouw updated and created tests for PHPUnit. Timo Tijhof fixed layout bugs in the Gadget manager, did some code review, and tested the migration of gadgets on a prototype.

Feature support

Mobile

  • Mobile Research [?]Mani Pande and Parul Vora consolidated all the research findings from Brazil, India, and the USA into one report. It’s currently being converted to PDF and wikitext to facilitate its publication.
  • MobileFrontend [?] — We quietly launched user login alongside better support for tiered JavaScript. We also fixed long-standing issues like the locked viewport, and we updated image description pages. Finally, we deployed HTTPS support on mobile for Wikipedia, with plans to enable it for sister projects soon.
  • Android Wikipedia App [?] — Several release candidates were released over the month and we’re nearing completion of the first version of the app, thanks to developers Yuvaraj Pandian and Brion Vibber. Nightly builds are available for testing.
  • WikipediaZero — We began work on the infrastructure for zero-rated Wikipedia access. Next month, we’ll start testing with one of our partners to work out the kinks of giving users free data access to Wikipedia.
  • GPS Storage/RetrievalMax Semenik joined the mobile team and began prototying an API to store and retrieve GPS coordinates on our wikis. This will be a critical component of the mobile projects; it will replace our existing use of GeoNames.org and can also supplement GeoHack.
  • Featured Article RSSMax Semenik built the first version of an extension to expose featured articles, In the news, and other main page content so that our partners can better re-use our data.

Special projects

Fundraising support

  • 2011 Fundraiser [?] — The DonationInterface extension underwent enhancements to tighten up security. Support was also added for monthly recurring donations for credit cards through our new payment processor, GlobalCollect, and we are working on automating the processing of recurring payments to our instance of CiviCRM. We built custom mass-mailing scripts to e-mail about 1 million past donors to encourage them to donate again. The ContributionReporting extension was enhanced by storing aggregated data in their own tables and updating them periodically, to eliminate the cache stampede problem uncovered last month. We added support for automatic notification of non-credit card payments from GlobalCollect, which allows us to automatically record donor and donation information in our donor database.

Offline

Platform Engineering

MediaWiki Core

The “MediaWiki Core” team was featured on the Wikimedia Tech blog this month.

Wikimedia analytics

  • Wikimedia Report Card 2.0 [?] — The reportcard 2.0 was moved to the Labs environment, and its source code centralized. The back-end and front-end code of stats.grok.se was rewritten and is being deployed to Labs as well. A renewed effort is expected as new employees come on board in January.

Technical Liaison; Developer Relations

Future

The engineering management team continues to update the Software deployments page weekly, providing up-to-date information on the upcoming deployments to Wikimedia sites, as well as the engineering roadmap, listing ongoing and future Wikimedia engineering efforts.


This article was written collaboratively by Wikimedia engineers and managers. See revision history and associated status pages. A wiki version is also available.

Archive notice: This is an archived post from blog.wikimedia.org, which operated under different editorial and content guidelines than Diff.

Can you help us translate this article?

In order for this article to reach as many people as possible we would like your help. Can you translate this article to get the message out?

5 Comments
Inline Feedbacks
View all comments

You should do a ‘year in review’ article for the technology. Would be much appreciated, to show that progress is indeed happening.

No word on LiquidThreads, is it dropped altogether?

LiquidThreads dropped ? I hope not ! It looks great, and, a contrario, the current talk way in Wikipedia is really bad, really not made for discussions.
Nicolas

Ioannis: The Wikipedia Signpost traditionally publishes a “year in review” issue, which includes an article about technology. They’re an independent community newspaper, so they should provide an unbiased progress report.
Trazeris and Nicolas: The status of LiquidThreads was recently brought up on the wikitech-l mailing list; you might want to read the responses given there.

@Guillaume, thank you for the link