Cut Off

Last week’s Comcast outage left locals in the lurch, some consider building redundancy

by

The full economic impact of a daylong internet outage last week in Santa Fe and Los Alamos might never be known, but local businesses are still reeling from the tough day. Some are looking to create their own internet redundancies to avoid future headaches.

Xfinity customers across Santa Fe and Los Alamos lost internet for most of Wednesday last week after a dump truck pulled down lines at the intersection of Cerrillos Road and Richards Avenue. Internet service and, in some cases, phone service went down around 12:30 pm and wasn’t fully restored in Santa Fe until 10 am the next day (Los Alamos regained service by 11:30 pm on July 20).

Police cited the dump truck driver for “inattention,” city police Chief Paul Joye tells SFR.

Comcast—Xfinity’s parent company—doesn’t release the number of customers affected by outages, spokesperson Julianne Phares tells SFR. But she says the incident did affect all Xfinity customers and all Xfinity services in Santa Fe and Los Alamos, including suburbs like Eldorado.

The outage threw much of the city into chaos as residents using Xfinity for work, communication and more flocked to coffee shops and public libraries to try to get service. Repairing the lines wasn’t simple.

“Repair crews hung new fiber and coaxial lines, splicing new sections to replace the damaged portion of the lines,” Phares tells SFR in an email. “The damage to these lines was significant and required multiple crews working in tandem to repair. This is the main fiber line for Santa Fe and Los Alamos.”

For some, this highlighted the precarious nature of the city’s reliance on a handful of internet providers.

“It really draws back the curtain on the vulnerability of our society,” says Brian Williams, emergency management director for the City of Santa Fe Office of Emergency Management. “We’re all just one car accident away from not being able to pay the rent.”

Williams says nothing major was affected within city operations (city offices are on CenturyLink, city IT director Manuel Gonzales tells SFR) but adds that for many, an outage like last week’s is an emergency.

“Obviously somebody who’s trying to pay their bills and they can’t get on the internet; that clearly is an emergency to them,” Williams says.

State of New Mexico Department of Information Technology spokesperson Renee Narvaiz agrees:

“DoIT is concerned about any outages and even the potential for outages,” Narvaiz writes in an email. “A brief outage can become an emergency. Consider a physician who cannot access a patient record securely, or a person who is unable to make a phone call for emergency services.”

She tells SFR that the department has multiple internet service providers such as Lumen, Plateau, Windstream and Unite Private networks, and that executive state buildings are equipped with internet access and interworking Layer 2 circuits to protect them from outages.

“Diverse path[s] into state internet services minimizes down time for constituents to access state resources,” she writes. “DOIT encourages additional ISP players to expand services across NM for a healthy delivery of broadband services to NM constituents at an affordable price.”

But that kind of redundancy isn’t an option for everyone.

Peter Wargo, principal systems architect for the National Center for Genome Resources, looked into various service options including local providers and a 5G T-Mobile hotspot when he moved into his home, but none except Comcast could offer what he needed.

“We’re pretty much shot for options,” Wargo says. “There’s nobody else here.”

He works from home, and has several smart home features that went down during the outage, like lighting. Wargo is also visually impaired, and relies on voice-driven service and online readers. While he has some tools that work offline, losing connectivity “takes away some of my independence,” he says.

“We have so many limitations that are artificial—the technology exists to do this right, and do it at reasonable cost. But for some reason it’s not happening.”

For Cliff Feigenbaum, founder and publisher of sustainable business and impact investing online publication GreenMoney.com, the outage was difficult to manage.

“In the middle of a work week, in the middle of a work day, the cost to my productivity was immense and across Santa Fe it was incalculable,” Feigenbaum says. He estimated Wednesday’s outage cost him thousands in possible ad revenue as well as media attention during GreenMoney’s 30th Anniversary year.

The economic impact of internet outages like Wednesday’s hasn’t been analyzed in Santa Fe, says spokeswoman Liz Camacho of Santa Fe’s Office of Economic Development. But according to a 2015 estimate by Massachusetts-based consulting firm IDC, internet downtime can cost small to midsize businesses $137 to $427 per minute.

“Being this technologically vulnerable that one supposed accident cut off the city—the millions of dollars of business that was affected—is a serious matter,” Feigenbaum says.

Albert Catanach, president and CEO of locally owned and operated internet service provider NMSurf, says he always recommends a backup connection for businesses that rely heavily on the internet. (Neither NMSurf nor competing local provider Cybermesa lost service when Comcast did.)

“The key to an outage is to have redundancy and resiliency,” Catanach says.

Internet redundancy means having a secondary connection that runs on a different path than the primary connection. Then, if primary service goes out, the secondary connection can kick in. However, Catanach says, building resiliency is difficult in Santa Fe:

“A lot of other states like our neighbors, Texas, Colorado, Arizona; they don’t have these problems as much because they don’t have the regulatory problems that we have here.”

Among these, he cites New Mexico Department of Transportation rules governing rights-of-way that pose problems for providers, and aesthetic regulations in Santa Fe around putting up Wi-Fi towers.

“In other cities, other states, they allow it because they understand that this is an economic boon for a city,” Catanach says. “I’m an aesthetic person too, but it’s a trade-off. Do you want better internet, or do you want mountain views?”

The cost of getting a secondary internet provider would also be prohibitive for many Santa Feans, but Catanach says it’s a compromise worth considering: investment in internet redundancy, or lost revenue during an outage.