Earlier this year, members of the Wikimedia Technical Operations Labs team and members of the Community Tech Tool Labs team merged into the Wikimedia Cloud Services (WMCS) team.
In this post, we outline what the new team is responsible for, what tools and projects fall under their umbrella, why and how the rebranding has taken place, and how you can learn more about products and services offered by the Cloud Services team.
What do you do, Wikimedia Cloud Services team?
The WMCS team focuses on four distinct areas:
- Providing a stable and efficient public cloud hosting platform for technical projects relevant to the Wikimedia movement.
- Developing, creating and maintaining services that empower the creation and operation of technical solutions to problems of the Wikimedia movement.
- Providing public, simple access to content and data produced by the Wikimedia projects to empower new technological solutions.
- Delivering technical and community support for users of the products.
What does that mean?
This new team is now in charge of Wikimedia Cloud VPS (formerly known as Wikimedia Labs), Toolforge (previously known as Tool Labs), and Data Services (which includes Wiki Replicas, ToolsDB, Wikimedia Dumps, Shared Storage, Quarry and PAWS.) The team works in partnership with the larger Wikimedia volunteer community to manage the physical and virtual resources that power the environment and provide technical support to volunteer developers and other Wikimedia Cloud Services users.
The new team is the latest in a long series of investments that the Wikimedia Foundation has made in supporting the technical communities who build tools to help the movement. The Wikimedia Labs project was started in 2011 to create an OpenStack powered environment where volunteers could become involved in helping the Technical Operations team. Over the past six years, the Foundation has committed more people and resources to these products and platforms. The scope has expanded beyond the initial vision to also include Tools developers, MediaWiki and application testing, analytics, and academic researchers. Our focus is increasingly on supporting our volunteer contributors and finding ways to attract more volunteers interested in making technical contributions to the Wikimedia movement.
What can I do with the Wikimedia Cloud VPS and Toolforge, and how do these platforms work together?
Toolforge is a shared hosting and platform as a service (PaaS) environment for volunteers who want to run bots, web services, cron jobs, or one-time jobs. Members create Tool accounts, which allow shared access by multiple maintainers to develop, deploy, and operate their tools. The Tool’s user account can use our Grid Engine job scheduling system or our Kubernetes container deployment cluster to run their code. The platform also provides easy access to the Wiki Replica databases and other Data Service products.
The Quarry and PAWS projects are trying to make using Cloud Services even easier than the Toolforge PaaS. Quarry lets anyone who has a Wikimedia user account run SQL queries against the Wiki Replica databases from the comfort of their web browser. PAWS is also available to all Wikimedia users and provides a platform for creating and running Jupyter notebooks or using Pywikibot from a command shell accessed from a web browser.
Wikimedia Cloud VPS is an infrastructure as a service (IaaS) product which uses OpenStack to provide virtual machines (VMs) for over 200 volunteer, affiliate, and Wikimedia Foundation staff managed projects. These projects include Toolforge, Beta Cluster, VMs for Wikimedia’s continuous integration system, and many others.
One of the unique features of Cloud Services is the Wiki Replica databases. These real-time replicas of the public metadata from the Wikimedia production wiki databases allow our users to perform a wide variety of data analysis.
The community has built hundreds of tools using these services. Some include:
- Vandal fighting tools such as ClueBot NG, which uses community trained models to detect and revert bad faith edits very soon after they are made.
- Copyright checking tools such as Copypatrol which provides a workflow for analyzing edits flagged by EranBot.
- Dashboards and lists of all kinds including the Wiki Education Dashboard, Wikidata Todo, and the shortest articles on Portuguese Wikipedia.
- Tools for translating content like Wikipedia page translation and Not in the other language.
- Tools for getting more content into Wikimedia Commons, such as flickr2commons and video2commons.
- Tools for exporting content like WSexport which converts Wikisource articles to EPUB documents for offline reading.
- Bots that do all sorts of things to help maintain the sites including Listeria which maintains list pages based on Wikidata
Why are you changing the names to Cloud VPS and Toolforge?
As we outlined during the community consultation process, the effort to rebrand the Cloud Services Team was designed to reduce confusion and ambiguity around the many projects with “Labs” in their names and to clear up confusion around what the word “Lab” actually meant. The OpenStack cloud and tools hosting environments maintained by WMCS have become more and more important to the Foundation and the on-wiki communities. We wanted the rebranding effort to both raise awareness of the existence of the OpenStack cluster and the shared hosting/platform as a service project, and to make clear that these projects are not experimental and are in wide use across the movement.
What is being rebranded and where can I see these new names?
A number of communication channels such as email listservs, IRC channels, and Phabricator boards are being renamed to ensure consistency. In addition, updates will be made on wiki, as well as at the domain and infrastructure levels. You can see all of the planned changes, and/or consult the list here:
- * “Tool Labs” has been renamed to “Toolforge”
- The name for our OpenStack cluster was changing from “Wikimedia Labs” to “Cloud VPS”
- The prefered term for projects such as Toolforge and Beta Cluster running on Cloud VPS is “VPS projects”
- “Data Services” is a new collective name for the databases, dumps, and other curated data sets managed by the Cloud Services team
- “Wiki replicas” is the new name for the private-information-redacted copies of Wikimedia’s production wiki databases
- No domain name changes are scheduled at this time, but we control wikimediacloud.org, wmcloud.org, and toolforge.org
- Toolforge and Cloud VPS have distinct logos to represent them on wikitech and in other web contexts
How can I learn more?
Watch the hour-long Introduction to Wikimedia Cloud Services (or YouTube), read about Cloud Services on wikitech, and look at the Annual Plan workboard on Phabricator.
Bryan Davis, Engineering Manager, Wikimedia Cloud Services
Thank you to Melody Kramer from the Communications Team for helping us with this post.