MySpace Outage Pinpointed at LA Telecom Building
25th July, 2006
The power outage that knocked MySpace offline was the second major failure in the past year at the telecom building in Los Angeles where MySpace houses much of its operations. The downtime at the Garland Building (1200 W. 7th) left MySpace users wondering how one of the web's busiest sites could go dark so easily. Adding to the headscratching was the fact that MySpace is a customer of Limelight Networks, a content delivery network that should (in theory) provide distributed caching and storage.
Other tenants at the Garland Building that were affected included DreamHost, which reported on the weekend power outages on its company blogs. The building lost grid power during a series of rolling blackouts that affected the Los Angeles area as California coped with stifling heat and record demand for electricity. The loss of power at Garland was blamed on the failure of a UPS system (uninterruptible power supply), which normally maintains power to equipment while generators start up.
Equinix said that reports that the outage may have occurred in one of its data centers were incorrect. "Equinix’s data centers were operating as normal over the weekend and this morning, providing consistent power to all customers," Margie Backaus, chief business officer of Equinix, reported in the comments section at GigaOm. "MySpace operates in Equinix data centers, but it also hosts operations in other data centers where we understand the incident occurred." MySpace recently announced that it would lease space in the new Equinix center in El Segundo.
The Garland Building suffered a prolonged power outage on Sept. 12-13 2005. The downtime was documented in a follow-up report to customers by Media Temple, another hosting tenant. The building lost grid power when a Los Angeles Power and Water Department (PWD) employee accidentally cut a power line, leaving 2 million customers in the dark. The Garland Building is equipped with five generators, but when two of them failed, the remaining three began shedding load and the facility lost power.
Some hosting tenants at 1200 W. 7th note that the building has infrastructure that would seem to make such outages unlikely. "The building power is supplied by two separate grids from the Los Angeles Department of Water & Power," colo provider IX2 Networks notes on its web site. "Unlike most buildings in downtown Los Angeles, this two-grid system gives the Garland Building a distinct advantage with the ability to switch to an alternate grid in the event one fails. The primary feed to the building is a preferred service that receives priority over all other grids in the event of an emergency."