On Aug. 14, alarms began to go off in Sprint's main network operations center at its headquarters in Overland Park, Kan. Normally, that's not a huge concern. Isolated trouble pops up now and again; that's one of the reasons network operations centers exist.

But that day was different. Before long, alarms were going off the way popcorn pops. “We didn't know what had happened, but we knew right away we had to convene the emergency management group,” said Jaimie Charlton, director of Sprint's National Technical Assistance Center.

What happened was North America's largest blackout ever, which wreaked havoc on seven states and two Canadian provinces.

Twelve hundred miles from Overland Park, Dave Rosenzweig, director of network operations in the east for Verizon Communications, was having a similar experience in New York City. A 33-year Verizon veteran who led the massive effort to restore the carrier's Lower Manhattan central office, heavily damaged in the World Trade Center collapse, Rosenzweig knows a major event when he sees one. He, like Charlton, just didn't know what kind of major event.

As major events go, there may be none more problematic for telephone service providers than a regionwide blackout affecting 50 million people. However, wireline carriers by all accounts came through the ordeal well—Charlton and Rosenzweig reported that neither Sprint nor Verizon lost a switch as a direct result of the outage—largely due to the manner in which the power systems at central offices are engineered.

Generally, switches and transmission equipment run off commercial power. When a failure occurs, enormous diesel generators automatically kick in. If those go kaput, huge 48-volt batteries immediately engage, capable of keeping a CO operating for about four hours.

One area of concern backup power systems don't address is the air conditioning systems needed to keep heat-sensitive switches cool. But that's a relatively small problem in the grand scheme, said Rosenzweig. “We can run without air conditioning for six to eight hours,” he said. “At [the facility damaged in the Sept. 11 attack], we ran switches in rooms where the temperature was 105 to 110 degrees for several weeks at a time.”

Although the diesel generators stand little chance of running out of fuel—Sprint continuously tops them off and keeps three days worth of fuel (about 10,000 gallons) on hand at all times, Charlton said — there is the matter of keeping them from overheating and damaging switches. Rosenzweig was forced to shut down two Manhattan central offices for precisely that reason.

“There was a problem with how the generators were configured,” he said. However, to keep customer impact to a minimum, the CO was turned down at 2 a.m. and brought back up by 8 a.m. the same morning.

Out of four generators, Sprint had one fail at a switch center in New York City. Though the failure didn't affect Sprint's ability to provide service to customers, the emergency response team decided that a replacement generator needed to be deployed. Under less intense circumstances, that wouldn't have presented much of a problem. But New York was gridlocked.

If anything good came out of Sept. 11, it is that the federal government is eager to help in times like these. A quick call to the Department of Homeland Security got Sprint the police escort it needed to get a replacement generator through the roadblocks and teeming masses that clogged New York's streets.

Of course, large telcos such as Verizon and Sprint have advantages in this area that smaller providers don't. For instance, zoning laws in New York City restricted General Telecom, a long-distance switch provider that shares space in the carrier hotel at 60 Hudson Street in Manhattan, to a scant 275 gallons of diesel fuel on hand at any given time.

That didn't last long during the blackout period. Compounding the problem was that once the building was evacuated, management wouldn't let anybody back in, including diesel refuelers dispatched by the provider. “That made us realize we can't rely on refueling,” said Lee Story, General Telecom's executive vice president for network services. Fortunately, General Telecom had adequate battery backup and is now looking at ways to beef up that option.

Wireless carriers did not fare as well.

Much of the trouble stemmed from factors having nothing to do with power. In the immediate aftermath of the blackout, millions of people turned to their wireless phones even though the redundancy built into the wireline networks kept them operating nearly at full capacity. Why? Because workers generally are immediately evacuated from big city high-rises when the lights go out for an extended period.

The heavy call volume eventually overwhelmed the network, and wireless carriers reluctantly resorted to call blocking to varying degrees. For instance, Sprint blocked about 25% of wireless calls leaving New York City at the height of the crisis, but blocked no incoming calls, said Charlton.

Wireless power system engineering also played a significant role in the spate of dropped calls and “fast busies.” Unlike wireline systems, everything on a wireless system runs off batteries: the handsets, the switches and the cell sites. Commercial power flows into a charging unit called a rectifier that keeps the battery at full capacity, much like an alternator keeps a car's battery charged. Should the commercial power fail, the rectifiers automatically kick to a backup generator.

Most if not all switch sites have backup generators, but many cell sites do not. There are several reasons for this. Of the tens of thousands of cell sites nationwide, many are in residential areas where zoning laws prohibit huge generators containing thousands of gallons of messy, smelly diesel fuel. Also, some cell sites are housed in buildings where logistics preclude the deployment of a generator.

But the lack of generators in many spots also has something to do with the mindset of wireless carriers, said Robert Sanchez, vice president and chief technical officer of industry consultancy inCode. “In metro New York City, the national wireless carriers have 900 to 1300 sites each. It's a cost issue,” he said.

John Celantano, president of industry analyst Skyline Marketing Group, agreed: “Wireless carriers have put less emphasis on backup power partly because they have been in growth mode and partly because they don't have the lifeline responsibilities that wireline has.”

The lack of backup power became an even bigger problem as the blackout dragged on because battery capacity generally ranges from two to eight hours. Eventually, thousands of sites ceased operation. In addition to blocking calls, wireless carriers compensated by rerouting calls to sites where adequate redundancy was in place.

“New York City is very densely populated, and there's a lot of cell site overlap,” said a spokeswoman for Verizon Wireless.

The downside of this technique is that forcing a site to handle thousands more calls results in a weaker signal for all of the calls being transmitted by that site. A carrier could turn up the radio amplifier to compensate, but that would cause the battery to drain faster. But in the end, this presented a better solution than imposing additional call blocking.

Sanchez suggested that carriers should consider beefing up strategic cell sites in their networks to be able to handle even more rerouted calls, as well as degrading the quality of the signal should they ever face another power outage of this magnitude. Voice calls typically cycle at 8 kb/s, and a reduction to 6 kb/s hardly would be noticed, he said. Such a reduction would ease the strain on the site, which would in turn reduce the power draw on the battery.

“We can tolerate 4 kb/s and still understand it,” Sanchez said.

New and better energy sources are another option. The automotive industry currently is testing and refining the use of fuel cells, which run on liquid hydrogen, and Sanchez thinks they might be a viable option for wireless carriers someday. But Celantano isn't convinced. After all, the Hindenburg was filled with the stuff, and we know how well that turned out.

“A telco guy is going to see pure hydrogen tanks sitting there and is going to get a little antsy,” Celentano said. “They get antsy just being around batteries.”