Power corrupts. Power failure corrupts absolutely.

We're currently recovering servers from a power failure in our colocation facility. This means backing up 170gb of database on several servers and running recovery. Back soon...

Where's my Wikipedia?

In a hurry to look something up? Try a google search for the page you need and view it from the google cache...

Wikimedia fundraising drive

The Wikimedia Foundation fundraising page is available as a static page for the time being.

Update 22:32 PST: moved temporary location to our main servers.

What happened?

At about 14:15 PST some circuit breakers were tripped in the colocation facility where our servers are housed. Although the facility has a well-stocked generator, this took out power to places inside the facility, including the switch that connects us to the network and all our servers. (Yes, even the machines with dual power supplies -- both circuits got shut off.)

What's wrong?

After some minutes, the switch and most of our machines had rebooted. Some of our servers required additional work to get up, and a few may still be sitting there dead but can be worked around.

The sticky point is the database servers, where all the important stuff is. Although we use MySQL's transactional InnoDB tables, they can still sometimes be left in an unrecoverable state. Attempting to bring up the master database and one of the slaves immediately after the downtime showed corruption in parts of the database. We're currently running full backups of the raw data on two other database slave servers prior to attempting recovery on them (recovery alters the data).

Update 19:20 PST: We have at least one database server with intact data. When we have a second up and running, we'll be able to put the site back online in read-only mode as we continue.