Uploads temporarily offline for site fix (done!)

Image (1) 120px-Gnome-face-sick.svg.png for post 3751Uploading and generation of new thumbs will be temporarily disabled on Wikimedia sites while we patch & reboot the server to fix the performance issues we’ve been seeing.
We hope to be done within a couple hours (by 22:00 UTC or so — 3pm PDT), but it could run shorter or longer.
Rough procedure for the curious:

    • Take image thumbnailing servers offline
    • Disable uploads
    • Unmount file server from web servers
    • Patch & reboot file server: rebooted – 21:00 UTC
    • Remount file server on web servers – 21:09
    • Put image thumbnailing servers back online – 21:12
    • Re-enable uploads

<- Done 21:18!
With the kernel fix, the file server should now behave better. We’ll then be able to continue our more leisurely migration of thumbnail files to another server, freeing up disk space on the primary box.
Updated 20:20 UTC: Added our hoped-for ETA
Update 20:44 UTC: A side-effect of taking the image server offline broke account creation and some editing which triggers an anti-bot captcha. Have switched to the simple captcha mode which doesn’t use images for now.
Update 20:56 UTC: Just noting that this affects <math> and <timeline> rendering as well. You may see some math rendering errors until we’ve completed; sorry!
Update 21:12 UTC: File server is back online and uploads are re-enabled. So far so good!
Brion Vibber, Lead Software Architect

Archive notice: This is an archived post from blog.wikimedia.org, which operated under different editorial and content guidelines than Diff.

Inline Feedbacks
View all comments

Is there any ETA yet?

Provisionally by 22:00 UTC (3pm PDT). Could run shorter or longer, depending how things go — hopefully the patching will run smooth and it’ll be fine once it’s booted back up! Otherwise we’ll have to jump through some hoops to move thumbnails to another box and regenerate them — which would still slog down image display for a while.

I don’t see anything above that says TeX will stop working. One can’t do anything on Wikipedia without TeX (except things that require no “displayed” mathematical notation; and of course sometimes “inline” as opposed to “displayed” notation uses TeX as well).

Hope you can fix this soon.

Oh: There’s the note dated 20:56 that mentions that.

Ok, we’ve patched the server which should resolve the super-slowness bug for now while we continue rearranging servers to keep more breathing room on the disks. Uploads are back online, and math should be working again.

IPv6 (2620:0:862:1::80:2 in my case) seems to have stopped working after this upgrade?

It’s still quite random if a thumbnail is displayed or not… 🙁

Yeah, the patch has made less of a difference than we’d hoped for. 🙁 We’re going ahead and bulk-moving thumbs out …