Database errors on most Wikipedias

At 10:57 UTC, the master database server for s3 (the cluster that holds most of our wikis) had a full disk and stopped writing. For this reason it was no longer possible to edit these wikis. The larger wikis live on separate clusters and were not affected.
After switching to another master database, all wikis are back up and editable as of 12:02 UTC. A few edits that were made during the incident may have been lost.

Archive notice: This is an archived post from blog.wikimedia.org, which operated under different editorial and content guidelines than Diff.

20 Comments
Inline Feedbacks
View all comments

Heh. That’s a new one :).

Hope you get back up soon!

working again now

Don’t we do capacity planning? And by “we” I mean “you”

I think you should migrate the other large wikipedia (spanish wikipedia) from s3…

There are report on sr.wp that pages edited around this time got wrong text on them.
E.g.
http://sr.wikipedia.org/w/index.php?title=%D0%97%D0%B0%D0%B3%D0%B0%D1%92%D0%B5%D1%9A%D0%B5_%D0%B2%D0%B0%D0%B7%D0%B4%D1%83%D1%85%D0%B0&oldid=3540110
This is a text of another article (not what user submitted). Also note that on the history page the byte count is wrong.
r.

I think you also need to consider migrating the English Wikinews off there too. I notice en.ikipedia and en.wikiquote aren’t on there, nor is commons. And what could possibly be up with having meta.wikimedia on s3? I think there’s a priority review needed there.

Robert Stojnic :

There are report on sr.wp that pages edited around this time got wrong text on them.

E.g.

http://sr.wikipedia.org/w/index.php?title=%D0%97%D0%B0%D0%B3%D0%B0%D1%92%D0%B5%D1%9A%D0%B5_%D0%B2%D0%B0%D0%B7%D0%B4%D1%83%D1%85%D0%B0&oldid=3540110

This is a text of another article (not what user submitted). Also note that on the history page the byte count is wrong.

r.

This is very strange. It looks like this revision somehow got linked to the wrong page. This is what I get when diffing this weird revision against the latest revision of [[Законодавна_власт]]:
http://sr.wikipedia.org/w/index.php?title=%D0%97%D0%B0%D0%BA%D0%BE%D0%BD%D0%BE%D0%B4%D0%B0%D0%B2%D0%BD%D0%B0_%D0%B2%D0%BB%D0%B0%D1%81%D1%82&action=historysubmit&diff=3540110&oldid=2898320
Mind filing a Bugzilla bug about this?

Thor Malmjursson : I think you also need to consider migrating the English Wikinews off there too. I notice en.ikipedia and en.wikiquote aren’t on there, nor is commons. And what could possibly be up with having meta.wikimedia on s3? I think there’s a priority review needed there. There seems to be the misconception out there that a wiki being on s3 somehow means it has a lower “priority” or that s3 somehow is structurally less stable than the other DB clusters. None of these things are true. If I wrote anything suggesting otherwise, I apologize for not being clearer. About… Read more »

god bless wiki 🙂
no idea Y it is laking on Storage 🙁
tie-up of Google and WikiMedia might solve such problem ! 😀

catrope :

Mind filing a Bugzilla bug about this?

Original reporter submitted it as:
https://bugzilla.wikimedia.org/show_bug.cgi?id=24726
r.

catrope There seems to be the misconception out there that a wiki being on s3 somehow means it has a lower “priority” or that s3 somehow is structurally less stable than the other DB clusters. None of these things are true. If I wrote anything suggesting otherwise, I apologize for not being clearer. It’s ok, Roan, no biggie – just when you posted yesterday in the -tech channel and on here, iirc you made a reference to small wikis, which gave me the impression that the ones I mentioned (meta, en.wikinews) were considered “small” in relation to others and therefore… Read more »

I suggest using http://en.wikipedia.org/wiki/Nagios for disk space monitoring – you can even make it send sms alerts BEFORE the disk gets filled, and it works ! 🙂

Przemek :

I suggest using http://en.wikipedia.org/wiki/Nagios for disk space monitoring – you can even make it send sms alerts BEFORE the disk gets filled, and it works ! 🙂

We do use Nagios, but it looks like somehow no one noticed the almost-out-of-space alert.

The deletionists are right! Wikipedia *is* running out of space, it’s just not the English WP which is! =O
(I actually can’t believe no one else has said anything to this effect yet)

This type of problem can actually be anticipated since space disk can be monitored all the time. I hope wikipedia would have funds to upgrade its servers and data storage facility.

Thank’s for you’re support.

Thanks for letting us all know.

Hey,
Is wikitravel.org also down? I am unable to access the websites from a long time.

Harshith :
Hey,
Is wikitravel.org also down? I am unable to access the websites from a long time.

Wikitravel is not hosted by the Wikimedia Foundation. Any server issues on either site are not connected to the other.