This is the third in a series of special reports on business continuity.

Disaster recovery has been very much on people’s minds a year after Hurricane Katrina, but for AT&T, disaster recovery and business continuity have been an ongoing challenge that isn’t associated with any single event but with many.

And in the wake of Katrina, the person most responsible for AT&T’s network recovery plan thinks the industry in general needs to address some concerns. First among them is electrical power.

“The thing that hits us most is loss of power,” said Robin Beinfait, senior vice president of global network operations for AT&T. “We as a country and globally need to do more about understanding alternative sources of power.”

Second, there is the issue of vulnerability to cyber security threats in the wake of a physical network disaster.

“Katrina was a very physical domain--but we could also have been attacked logically,” she said. “When you have lost power and access, it’s natural to focus on the physical aspects of getting your network back. But that can be a distracter. When you get a network back up on impaired resources, your logical security parameters are also weakened. We have to be more careful in dealing with that. Our customers are the same way--they are so eager to get their applications up online, their firewall issues are not in that same lineup. They need to be just as concerned with their firewall and their security in the post-disaster environment. That is when everybody’s guard is down.”

As much as possible, AT&T doesn’t let its guard down, Beinfait maintains. The telecom giant first designed its business continuity plan in 1991 and has spent $300 million since to implement it. Whenever there is a major event in the U.S., whether weather-related or scheduled, such as the Super Bowl or a national political convention, there are elements of AT&T’s “network on wheels” assembled somewhere nearby, she said.

“When we see a hurricane coming, we’ve planned for it,” Beinfait said. “My team has gone looking and tracking the footprint, and set up diverse routes to get into areas. We run simulations--if I were to lose this area, I’ve got to have 27 trailers, and if we lose another area, we need more. I stage around the effected areas. I stage around the Super Bowl, political conventions, the world soccer event. It doesn’t matter whether it is an American event, we stage around the Olympics as well--anything we think might have potential as a target. This is not just a disaster recovery for the U.S. domestically, but we have plans globally.”

From four separate and secret locations in the U.S. and one in the U.K., AT&T maintains 150 network equipment trailers and 350 power-equipment trailers that constitute its network on wheels. That gear can be driven or flown into disaster sites as needed, Beinfait said. But it is only the most visible part of the AT&T plan.

“We operate off some very sophisticated complex proprietary software tools,” she said. That software enables AT&T to replicate network nodes either physically or logically in the event that one of its 3500-plus nodes or 6000 locations is knocked out for any reason.

“Every year we go through a cycle of weather events--ice storms, tornados, landslides, hurricanes, any of these type of things. We know when they are coming about,” Beinfait said. “We make sure, as we design our network or network footprint, that we have the diversity to replace our network. We have to continue to reinforce that in the embedded design.”

A 20-year AT&T veteran whose only move away from operations has been an occasional stint at AT&T Labs, Beinfait said the company does learn something with each disaster. With Katrina, for example, “we realized some of our own ground communications can be improved. We brought in a wireless walkie -talkie type of service, so ground management can have more effective communications.”

AT&T also brought in broadband wireless gear and will probably use it more extensively in future outages of a serious scope, she said. As technology evolves, so does the disaster recovery plan.

AT&T put its disaster recovery investment on display at TelecomNext last March, and conducted VIP tours for attendees including regulators such as FCC Chairman Kevin Martin. The FCC is currently conducting proceedings regarding disaster recovery.

Beinfait said AT&T continues to work with its business customers to bring them up to speed, helping them set up their own disaster recovery and business continuity plans.

“Business continuity is to be able to keep your business operating not only to survive but thrive during a disaster – either logical or physical,” she said. “It’s making sure your design for your business had resiliency that your business mandates.”

Related Articles

Redefining business continuity (Part 1)

Disaster recovery: Should voice mail be free? (Part 2)

A TelephonyOnline podcast: Business Continuity