Gallery: Guns, Squirrels, and Steel: The Many Ways to Kill a Data Center
Ingrid Taylar01squirrel
Data centers go down all the time. It's a fact of life. But while we've all seen the internet error messages, we don't always hear what actually brought down the massive computing facilities responsible for serving up the web services we use every day. Two weeks ago, Netflix went down on a Friday night. The problem? A massive storm on the east coast and some Amazon backup generators that didn't power up in time to help. Here are few more root causes behind data center outages over the past few years. They include everything from lightning to squirrels to boredom. __Squirrels__ Squirrels may be the data center's enemy number one. Level 3 Communications [say that they accounted for an astounding 17 percent of all of their cable damage](http://blog.level3.com/2011/08/04/the-10-most-bizarre-and-annoying-causes-of-fiber-cuts/) last year. The thing is, squirrels are little chewing machines, and they love to chew right through whatever is in their way, even if it's a critical fiber-optic cable or (bad news for the squirrel) a live power line. That's what happened to Yahoo a couple of years back, according to Mike Christian, a director of engineering at Yahoo. " A frying squirrel did take out half of our Santa Clara data center two years back," he said, [speaking at the Velocity conference in Santa Clara, California last month.](http://www.youtube.com/watch?v=iO2z3ttlpi4&feature=player_embedded#!) As [Data Center Knowledge pointed out recently,](http://www.datacenterknowledge.com/archives/2012/07/09/outages-surviving-electric-squirrels-ups-failures/) squirrels account for an astoundingly large percentage of power outages. Steven Hebert, a wildlife specialist with Swat Pest Control in San Jose, Calif. says that most of the time the culprit is the eastern grey tree squirrel. These guys don’t hibernate or burrow like their ground squirrel cousins. How do you tell a ground squirrel from a tree squirrel? It's easy, Hebert says: "Clap your hands. If it runs up a tree, it's a tree squirrel." *Image: [Flickr/Ingrid Taylar]( http://www.flickr.com/photos/taylar/3646439040/)*
02gunmen
__Hunters__ It makes sense when you really think about it. You're hunting bear in the Oregon back woods, but pickings are slim. Suddenly, you see an insulator hanging from a pole up in the air. Maybe you say, "I'll bet you $20, I can hit it." "What people do for sport or because they're bored, they try to shoot at the insulators," Google engineering manager Vijay Gill [told a networking conference in Sydney a few years back.](http://www.itnews.com.au/News/232831,us-hunters-shoot-down-google-fibre.aspx) It turned out to be a real problem for Google's $600 million data center in The Dalles, Oregon. "Every November when hunting season starts invariably we know that the fibre will be shot down, so much so that we are now building an underground path \[for it\]," he said. *Image: [Flickr:State Library of South Australia](http://www.flickr.com/photos/state_library_south_australia/4539032589/)*
03lightning-0
__Lightning__ Storms are often bad news for data centers. Two weeks ago 80 mile-per-hour winds knocked out power at a pair of Amazon data centers in Virginia. When backup generators failed at one of them, the Amazon cloud [crashed for a number of customers, including Netflix, Pinterest and Instagram.](http://stag-komodo.wired.com/wiredenterprise/2012/07/amazon_explains) But even more dramatic is the good old lightning strike. That hit Amazon's cloud back in June 2009, when [a lightning storm took out servers]( http://www.datacenterknowledge.com/archives/2009/06/11/lightning-strike-triggers-amazon-ec2-outage/ ) after damaging one of Amazon's power distribution units. *Image: [Flickr/Andrew Rivett](http://www.flickr.com/photos/veggiefrog/2573076568/)*
04power-off
__Shiny Red Buttons__ We call it the Shining Syndrome. You spend days and days unboxing servers in a data center on a tight deadline. There's no natural light, little human contact, and always the hum of servers… that goddamned hum! This one comes to us from John Bumgarner, chief technology officer with the U.S. Cyber Consequences Unit. Back around 2000, he was working as a consultant with about 15 other geeks on a big Unix rollout at First Union Bank's data center in Charlotte, North Carolina. Bumgarner worked with a pretty junior consultant. Bumgarner didn't tell us the name of junior man, but it was junior's first job out of college, and he had the completely tedious job of unboxing servers and booting them up. Months went by, and there was pressure to finish the job. "The consulting firm that I worked for starting making some of their junior employees work overnight hours in the data center deploying new equipment," Baumgartner says. It was pretty boring work. While waiting for software installs to finish up, you couldn't watch television or even listen to music. So the unboxer took to poking around the massive data center. "For some unknown reason, this employee decided to press this red button on the data center to see what it was for." He says. "Of course, this button was the emergency power shutoff for the entire datacenter. " That was the end of the $30 million consulting gig. The bank's systems went offline, costing millions in lost revenue and lost productivity. Image *[Flickr/Richard Masoner](http://www.flickr.com/photos/bike/899973276/)*
05image-3
__Explosions__ It was a typical Wednesday afternoon on the 13th floor of the Shaw Communications building in Calgary, Alberta. Suddenly, [there was a bang, and then sprinklers kicked on,]( http://stag-komodo.wired.com/wiredenterprise/2012/07/explosion-shaw-ibm/) flooding the data center, and starting a data meltdown that the Canadian city is still trying to dig itself out of.
06burgler
__Thieves__ Danish internet service provider Nianet went down three years ago, when thieves cut a hole in its Taastrup data center's walls, snuck in, and then kicked in a door to get at the networking gear. Their goal? According to [this report in Jyllands-Posten,]( http://blogs.jp.dk/it-bossen/2009/07/02/us%C3%A6dvanlig-nedbrudsforklaring-fra-nianet/) they stole a bunch of networking cards, knocking Nianet offline for a bit. *Photo: [Flickr/Kojach]( http://www.flickr.com/photos/kojach/4026783814/)* Hat Tip: Knud Erik Højgaard
