Global Outage (cooling failure and DNS)

Due to an overheating problem in our European data center many of our servers turned off to protect themselves. As this impacted all Wikipedia and other projects access from European users, we were forced to move all user traffic to our Florida cluster, for which we have a standard quick failover procedure in place, that changes our DNS entries.
However, shortly after we did this failover switch, it turned out that this failover mechanism was now broken, causing the DNS resolution of Wikimedia sites to stop working globally. This problem was quickly resolved, but unfortunately it may take up to an hour before access is restored for everyone, due to caching effects.
We apologize for the inconvenience this has caused.
Update: Unfortunately, for many, this outage seems to have lasted longer than an hour. It appears that many ISPs’ DNS resolvers do not honor the so-called Negative Cache TTL that we send (1 hour), and instead use a longer value. We have circumvented this problem by renaming the affected DNS record to something else.
Update 21:32 UTC: Our SSL gateway, secure.wikimedia.org, was disabled due to overload issues, but is now back up.

Archive notice: This is an archived post from blog.wikimedia.org, which operated under different editorial and content guidelines than Diff.

97 Comments
Inline Feedbacks
View all comments

And so we remember Thursday, March 25, 2010 as the day every English speaking student failed their research papers.

Nice to see you getting back online!

How am I to study for my test without wikipedia!??!?

Who’s data center is that? Is it run by Wikimedia?

Damn, I have a paper due in an hour and a half…

Shit happens :-))
Nice to see you stil alive

All you slackers waiting until the last minute. If you fail to plan you should plan to fail

Given Wikipedia is free is it not our right to criticize when something fails.
The site is sorely missed and we stand that you solve this issue speedily.
Well done for everything you do to assist humanity.

For you needing to do research, just go to: archive.org and copy and paste the wiki url.

@1: not only english speaking ones…

Duh…use Google cache already…it has the wikipedia pages cached [w/o images of course, but something is better than nothing]!

EvoSwitch, I presume? Was it the innovative cold corridor solution that gave or what?

Bah, now what to do at work?

The following entries can be added to one’s local “hosts” file:
208.80.152.2 en.wikipedia.org
208.80.152.3 upload.wikimedia.org

I could load the site via proxy while it was down, but its working ok now.

Don’t use wiki for papers.

At 1824, I could not raise the server for the RSS feed.

@Mazen Abdel-Latif
Use a proxy if it still isn’t working for you, it worked for me on every proxy site I tried.

RSS problem resolved 1850 thanks

Well I donated $5 to them for that big fund raiser they had, so my whining is semi-legitimate, but I won’t, because wikipedia rocks. But really though I’m bored at work without wikipedia, please come back!

Thats the first time for me, and that’s a long time i.e. ever.
Wikipedia we love you. Its refreshing to NOT instantly get the answer to just about anything fact-based question you need to ask. It reminds us of life before Wikipedia and how valuable this resource is to us (connected) today.
A Thought Experiment and a bit of fun… is Wikipedia truly disaster proof, with 2012 looming and all that ?

Wikipedia had been down also here in Italy. 🙁
Good work…

Hi:
Wikipedia worked fine for me over the last few hours; maybe caching helped. Also, apparently the .m.wikipedia.org domains were accessible during the DNS outage.
Thank you for continuing to provide this valuable service!

Wiki comeee on I need u guys…………… lesson learned, move all dedicated servers to U.S and problem solved.

Geoffrey Hartwell :
At 1824, I could not raise the server for the RSS feed.

In 1824, I couldn’t even log on to those innerwebs….

If you use Google Chrome to access wikipedia, you can click “Access a Cached copy of Wikipedia”. When you click it, you can go to the address bar and input what you want to search (example, New_York_City) after the en.wikipedia.org/wiki/ part.
It looks like this: http://74.125.113.132/search?q=cache:http://en.wikipedia.org/wiki/New_York_City
It wont have pictures though. But who needs those? :p

Funny fact, Google redirect it’s china’s traffic to an unfiltered search engine and Wikipedia goes down.. Yeah, it’s just me being paranoid 😛

Puro Caino :
Funny fact, Google redirect it’s china’s traffic to an unfiltered search engine and Wikipedia goes down.. Yeah, it’s just me being paranoid 😛

Wow, I noticed that too. XD

Shit happens guys. Don’t worry, I’m sure the world won’t stop because the world’s best reference and direction-setter is down.

Hello, people! 🙂
Users: Don’t worry, let us trust.
Wikimedia people: I believe that many believe (me, one of them) you’ll do what is to be done.
Anyway, for possible future events (and more), it can be useful to get to know what is this situation like.
Thank you for your efforts, Wikimedia people! 🙂
Enrique
http://www.forvo.com/word/Enrique#es

Well, I guess we shouldn’t be throwing away all of our books just yet, should we?

[citation needed]

Please don’t bring tears in our eyes… please come back our dear dearest Wiki… we love you.

¡Hola, gente! 🙂
Usuri@s: No se preocupen, confiemos.
Gente de Wikimedia: Yo creo en que much@s creen (yo, uno de ell@s) en que ustedes harán lo que ha de ser hecho.
De todas maneras, para posibles eventos futuros (y más), puede ser útil el lograr saber cómo es esta situación.
¡Gracias por sus esfuerzos, gente de Wikimedia! 🙂
Enrique
http://www.forvo.com/word/Enrique#es

Learn to Google Cache people.

Jimmy :
And so we remember Thursday, March 25, 2010 as the day every English speaking student failed their research papers.

Congrats, you’re quoted on CNN.

it still doesnt work with the default dns server of my provider here in germany. when using open dns there is no problem

If you can not access it use CoralCDN coralcdn.org

just like every other person, i reeally need to wikipedia back online, i have a history test tomorrow about the russian revolution and i need to maintain my high average which was the result of previous revisios sessions on wikipedia.

Maybe, Learn to wait ISP’s DNS update.
However, it is live for a long time.

You can go to their mobile site
http://en.m.wikipedia.org/

Thanks for the update guys 🙂 I was afraid you were gone for good because of either hackers or lack of funding. I’m glad my fears were hollow.

overheating problem! is it already the global warming?

xeno :
The following entries can be added to one’s local “hosts” file:
208.80.152.2 en.wikipedia.org
208.80.152.3 upload.wikimedia.org

What IPs for ru.wikipedia.org and another?

Jimmy :
And so we remember Thursday, March 25, 2010 as the day every English speaking student failed their research papers.

I’m doing mine without Wikipedia now. A pile of textbooks and printed “literatures” has better effect against professors compared to Wikipedia 🙂
Get well soon, Wikipedia!

Well, it’s 4pm in Florida (where the supposed back-up servers were supposed to save the day). I can get to pages & some info, BUT it’s a mess. There is no navigation at top or left of the page.
TIP: Turn off pop-up windows if you’re in Safari. It will help get the pages uploading sooner.

Damien :

xeno :
The following entries can be added to one’s local “hosts” file:
208.80.152.2 en.wikipedia.org
208.80.152.3 upload.wikimedia.org

What IPs for ru.wikipedia.org and another?

Same one…
208.80.152.2 ru.wikipedia.org
You might also need –
208.80.152.118 bits.wikimedia.org

Just use google’s memory function. Works for the most popular pages at least

I second what Leon says…
Jimmy that comment was brill 🙂