Nobody notices when it's not broken: New database servers deployed

The Technical Operations team has just completed behind-the-scenes work that will likely never be noticed by our readers.
Our External Storage databases hold the text for every version of every wiki page; they have slowly grown over the life of Wikipedia and its sister projects. Ten years is a lifetime on the Internet, and the incremental changes that were made to our external storage system over that period, though appropriate at the time they were made, resulted in a setup that was a challenge to maintain and which was becoming unreliable.

Graph of query durationWe spent a few weeks analyzing all the various servers across which the page text data was spread, in order to gather it all together onto a single host. From there, it could be replicated onto newer, more reliable and higher performance hardware. Along the way, we found and fixed a number of inconsistencies to make the dataset more regular.

The deployment of the new hardware lasted a few days (as we moved things piece by piece) and was finished this past Monday with no fanfare. There was a brief (about 10 minute) period during which articles were unable to be edited while we switched writes to the new hardware. The end result is a barrage of small improvements, all of which together make for a happy TechOps team:

  • average query duration has dropped from about 15ms to around 8ms and the worst case from 576ms down to 60ms;
  • replication and failover processes are now well known and standardized;
  • total hardware used has dropped from around 30 servers to 8, now in two locations;
  • hosts no longer double up as web servers and database servers for text; dedicated servers are used for the database.

It’s a small victory in the battle against entropy, but an important prerequisite for carrying out our mission of providing unfettered and reliable access to the sum of all knowledge.
Ben Hartshorne, Operations Engineer

Archive notice: This is an archived post from, which operated under different editorial and content guidelines than Diff.

Inline Feedbacks
View all comments

Hi Guys,
I read with interest that Wikipedia is asking for donation. For goodness sakes, put some google ads on your site lah. Sure, there will be people clicking on the ads and you will earn money for your operations that way.
And it will be willing people doing that. Just going and asking for donations is gonna piss people off and they may not return to your site, think about it

Hello, James! Currently, Wikimedia doesn’t need advertising. So the bosses hear what Wikimedia users and editors think to act. And lots of us believe that having no advertising helps to show that the project is non-profit and totally neutral. I’m sure that lots of people would prefer to see small advertisements around the year over having a huge donation advertisement a few weeks a year. But that preference is weaker than the strive for indenpendency. I hope that you understood the decision. Good luck!

@James : I’ve been SO not pissed of by the donation request that I’ve just doned 20 bucks.The guys here just deserve it. I use Wikipedia all day long (I’m an IT consultant) and I know how all the people at Wikipedia need money to make things work. A donation every now and then for a reliable source of information available 24/7 with no commercial affiliation whatsoever, what can I say more ?

@James: I use MediaWiki for my own website project. Since I am so happy with the (free, light, fast, stable and properly maintained) software I am glad to donate a few bucks.
With ads, nobody would donate. Just [x] Jimmy away if he annoys you.

Well done!
on too many occasions good work goes unnoticed, thanks for letting us (me) know about this.
It looks like you charged into this well prepared and executed without a hickup. I’m quite jealous.
Seems like the donation money is going to good causes, and well spent.
All the best,
Moshe Eshel

Congratulations ops!!

@james wikimedia is an organization and putting google adsense is only for people who are greedy enough to make internet a juice. Btw congrats for new servers

Señores Wikimedia, estoy interesado en ayudarlos a ustedes, pero no tengo tarjeta de credito, ni tampoco cuenta bancaria, me gustaria hacerlo en efectivo, sera que pueden ayudarme?