Everyone Has a Service Recovery Plan Until They Don’t

Extensive communications and connectivity service outages in parts of Florida and North Carolina in the aftermath of Hurricane Helene highlight the fact that traditional network resiliency and recovery plans are sometimes worthless in major disasters.

The situation is reminiscent of Mike Tyson’s famous reply when asked if he was worried about Evander Holyfield’s fight plan. He said everyone has a plan until they get punched in the mouth.

This week, enterprises and people in Helene's path have figuratively been punched in the mouth. Any planning before the event to use redundant carriers, routes, and services was meaningless when wired and wireless infrastructure and equipment were washed away or blown down. Add in widespread power outages and service restoration is even more challenging.

What should be done? It is not the first time we've been through such an event. Some notable past events exposed problems with traditional disaster recovery and service restoration planning. In each instance, there were lessons learned, and changes were made.

A closer look at past service resiliency and restoration shortcomings

Resiliency and contingency planning are essential for the availability of essential communications and connectivity services. But those plans can easily be usurped when the unexpected happens.

That certainly was the case with service interruptions and restorations after the attack on the World Trade Center. Physical communications infrastructure was wiped out. Cellular service for most of Manhattan was cut off when the towers and the antennas atop them fell. Data centers in the towers and adjacent buildings were destroyed.

In the aftermath, cellular service providers deployed antennas in a more distributed manner. Many financial services organizations located in the area built secondary, mirrored, or backup data centers across the river in New Jersey.

The backup plan needs a backup

Those plans worked well until 2012, when Hurricane Sandy tore through the region. Lower Manhattan was plunged into darkness when a ConEd substation flooded, leading to a major transformer explosion. Power in the areas was out for days.

Unfortunately, the backup and secondary New Jersey data centers had their own problems. Many required IT staff to physically go to the secondary site and take action. The problem was that many roads washed away or closed due to flooding preventing staff from getting to the sites.

Once the roads opened, IT staff that had to report to the secondary locations faced another problem. Like lower Manhattan, much of the New Jersey shore also suffered a major power outage. At the time, Public Service Enterprise Group, the state’s largest utility, said it was the worst outage in the company’s history. No power in the region meant gas stations could not pump gas and the IT workers could not drive to the backup sites.

As a result, some companies implemented technologies that allowed remote management of their data centers. That was part of a larger industry trend, and it served many enterprises well during the pandemic.

An opportunity for satellite services

One lesson from Helene is that no place is safe. Asheville, NC, which suffered great outages, is inland. Yet, flooding in the town matched (or exceeded) that of coastal communities in the hurricane’s path.

Until recently, when terrestrial wired infrastructure was wiped out due to floods and cell towers were ripped out of the ground, there really was no alternative to maintaining communications and connectivity.

That situation is starting to change as enterprises (and users) look to the sky. Specifically, there is growing interest in the use of satellite services to support offices and users in remote locations not served by terrestrial broadband services. Those services can also play a critical role in disasters.

For example, FEMA is making use of Starlink broadband satellite services to support communications and high-speed communications in impacted regions. The agency said that it will deploy one Starlink per county Emergency Operations Center to ensure communications. On Monday, the White House said it had already deployed dozens of the Starlink systems, with over 100 more in transit, according to Reuters. 

More satellite alternatives emerge

In addition to the use of Starlink, 2024 has seen a major shift in industry thinking about the use of satellite services in general. Why? There are multiple factors driving the interest in and need for satellite services.

To start, many enterprises now need global connectivity to support remote offices and workers. The widescale use of IoT in logistics, supply chain applications, and more requires data collection no matter where a truck, container, or personal device is located.

Fear of other types of outages, particularly in the vulnerable international subsea cable network, is causing enterprises to look at satellites to provide alternate routes to carry their traffic in the event of an incident.

Numerous efforts have been made this year to meet these demands. In February, a group of satellite operators formed the Mobile Satellite Services Association to accelerate the availability of direct-to-device (D2D) services for businesses and consumers.

After successfully launching two prototype satellites late last year, Amazon is poised to begin launching satellites for its Project Kuiper, the company’s FCC-approved project that will include more than 3,326 low-earth orbiting (LEO) satellites.

A final word

Past incidents exposed weaknesses of traditional network resiliency and disaster recovery strategies. Some of these incidents have led to changes and the introduction of new technologies that provide higher availability and more resiliency.

Perhaps the devastation of Hurricane Helene will get enterprises thinking about non-terrestrial service alternatives to bolster their networks and reduce the impact of future disasters.