Webhosting news

One year ago, Hurricane Sandy pounded New York City and the entire U.S. East Coast. While many of Manhattan’s residents were evacuated to avoid the storm’s destruction, two of PEER 1 Hosting’s data center team members—Mike Mazzei and Jeff Burns—stayed at our facility in lower Manhattan to ensure everything went smoothly during the hurricane. We really didn’t expect the storm to have a significant impact on the data center. At the worst, we thought the facility might lose utility power, and Jeff and Mike would need to switch to the building’s backup generators.

We couldn’t have been more wrong. When the storm surge hit on the night of October 29, 2012, the basement and first floor of the data center quickly flooded. The servers are on the second floor, so they were safe from the flooding, but the building’s main incoming utility switchgear and fuel tank reside in the basement, and they were completely under water. There was a limited amount of fuel in the generator header tank on the 18^th floor to power the backup generators on the 17^th floor, and Mike and Jeff knew that it would only last a couple hours. We quickly realized that Sandy was much worse than any of us had anticipated, and we quickly had to come up with a new plan to keep the data center running.

The initial reaction was to set up for a controlled shutdown, but Mike asked for one more hour to find a solution and prevent any customers from being affected by the storm. When I talked to him next, he told me that other local PEER 1 Hosting employees, and even some of our own customers, had arrived at the data center to help. Together, they were carrying fuel up the stairs to the 17^th floor in a makeshift bucket brigade. I have to admit, I was impressed by this resourceful approach. It wasn’t a perfect solution—nor did one exist in a situation like this—but it worked, and the team was able to keep our servers fueled for days. Eventually, we were able to install a temporary fuel hose up to the header tank that feeds the generators on the 17^th floor. And when the storm eventually cleared, PEER 1 Hosting’s data center was one of the few in lower Manhattan with power.

Hurricane Sandy was not our first hurricane, and we’ve seen numerous other natural disasters at some of our other data centers throughout the U.S. This was a very different experience, though, and there was a lot to be learned from the unique challenges that Hurricane Sandy presented. For instance, we learned how to set up an effective bucket brigade, and our customers learned that we stand behind our mission statement of being the most human company on the web. We also learned how we can avoid having to be in that situation ever again, along with some best practices to prepare a data center for a looming storm.

Of these takeaways, one of the most important lessons is that there are some things that data center managers simply can’t plan for. For instance, while I would derive significant peace of mind from never putting a generator or fuel tank in a data center basement again, that simply isn’t feasible, as there are issues of structural support on higher floors. We would also pay much more for the space this infrastructure would take up owing to steeper rent rates on higher floors, even though the generators and fuel tanks are very rarely used.

With that in mind, data center managers can actually do quite a lot to prepare their facilities for large storms. Here are my top tips to prepare for hurricane season and, really, any other natural disaster:

1. Get sufficient staff in the data center before a storm hits. We were extremely lucky to have local Manhattan customers at the data center the morning after the storm to help with the bucket brigade effort. Without all those extra hands, we would have had to do a controlled shutdown of the data center. If you can’t “staff up” before the storm, think about how you’ll get additional support to the facility if it’s needed. For example, you may want to set up emergency response teams near your data centers. That way, if there’s an outage, fire, earthquake or other disaster, you’ll already have a team close by that is ready to help.
2. Don’t wait until you need roll-up generators or fuel hoses in your data center—get that equipment on site today. Also, think about other supplies that you might need if the data center team has to stay in the facility for an extended period of time. Especially in your storm preparations, stock up on flashlights, sleeping bags, water and non-perishable food so the crew can stay, fairly comfortably, as long as they need to.
3. Make sure contracts are in place for maintenance, fuel and any other services that may be needed during and after a storm. These contracts will ensure that you have the supplies and assistance you need to get downed infrastructure back up and running, and the fuel to keep it working. Without contracts, you’ll be last in line, and you’ll probably end up shutting down your data center.
4. Avoid relying on everything going according to plan. You can plan as much as you want, but it’s impossible to be completely ready for any natural disaster. It’s our policy to regularly test a data center’s critical infrastructure and stock extra supplies on site before a storm, but these precautions were insufficient for Hurricane Sandy. You, too, will likely have to operate on the fly and adapt your disaster procedures during a storm.
5. For service providers like PEER 1 Hosting, it’s important to also think about customers and how you can help them prepare for a potential natural disaster. Counsel customers on the risks of their critical infrastructure failing and offer to help set up a disaster recovery solution. Another option is to use a public cloud environment so that should one facility go down, customers’ applications will fail over to another data center. Remember to also keep customers apprised if something does go wrong. For instance, if and when a storm hits, keep customers involved and aware of the facility’s status, especially if you think you will lose power or need to shut down. We’ve found that initially emailing customers and then posting frequent updates about a facility’s status on our website is the best approach.

Looking back on Hurricane Sandy, it was a very rough recovery and one that I hope to never experience again. But I also believe that the lessons we learned have made us a stronger and more capable company, and I am very proud of what we accomplished for our customers.

Leading article image courtesy of NASA Goddard Photo and Video

About the Author

Ryan Murphey is VP Datacenter Operations at PEER 1 Hosting. With more than 13 years of experience in data center management and customer-service management in the hosting and network industry, Murphey currently directs the design, budget and capacity planning for PEER 1 Hosting’s 19 data centers across the United States and Canada.

The post Why Data Centers Are Now More Prepared Than Ever Following Hurricane Sandy appeared first on The Data Center Journal.