We know that even when your power is out, you want the internet up and running
Cruzio knows our customers need internet service no matter what the circumstances. And that’s been a big part of our ongoing infrastructure investment.
Our aim is 100% uptime, even when PG&E power is out.
In fact, we aim higher than that: we offer extra service to customers who don’t have power in their homes or offices. If your power is cut, we always want to offer the alternative of coming into our coworking space and using the internet here.
As the PG&E situation gets more challenging, we have to adapt in order to stay close to our goal. This blog describes the steps we’ve taken in the past and what we’re doing now.
We’ve Always Prepared for Outages
We’ve prepared well over the years. As we established our independent fiber optic-backed network, Cruzio bought our own office space in downtown Santa Cruz to house a generator and data center and we’re connected with multiple redundant fiber and wireless backhaul paths.If one of our hubs goes down in a power event, we can reroute traffic in several configurations. Because of this, we’re able to confidently guarantee service levels to enterprise customers — businesses who simply can’t operate without internet. We’ve even been able to come through with sudden demands for emergency internet for local facilities like the County Building.
Meeting the day we got the PG&E outage warnings
We’re proud of our outage response team: Jesus Lopez, Dan Thomas, Chris Frost, Adia Schamber Jones, Justin von Besser, Mark Hanford, Alison Lowenthal, and James Hackett (not pictured). With the rest of our dauntless staff, they keep our network and downtown headquarters up and running, serving thousands of customers.
We have dozens of generators and uninterrupted power supplies (UPSs) to provide power backup to our many facilities around Santa Cruz County and surrounding areas. Cruzio always carries spare UPSs so that in an extended power outage we can cycle them (charge one while deploying another). In an outage, we follow a schedule of rotation and replacement for our UPSs and for fueling and refueling generators. Even when PG&E power is on, we run our main generator once a week to ensure it’s working properly.
We’ve Got a Great Team
Our team, mentioned above, is a crucial part of our uptime efforts. They’re the ones up at 3 am, driving to the top of Loma Prieta in bad weather, making sure the power is on and the internet’s flowing.
And when you add in the extra issues our customers experience in emergencies, every member of our staff contributes. We may not return your call or update your ticket as quickly when we’re in the middle of an outage, but we are listening to and addressing the issues you describe. Please check our network status page or call in for the latest news.
Once, when the elevator power was out, our Director of Technology and Infrastructure, Chris Frost, carried a UPS up 5 flights of stairs. By the time he got to the top and plugged in the equipment, the power had returned. He laughs about it. But he made sure we were ready for an extended failure.
But Now We’ve Got a More Serious Situation
Cruzio has been well prepared for the way things have worked up until the last few months. Generally, in our experience, we’ve had power outages for 3 or 4 hours, or perhaps half a day. The outages have been unplanned, occurring in a limited area, and PG&E has fixed them quickly — often even before their own projections. We are more than ready for that type of power outage; it can be challenging, but we have the equipment necessary to handle it. But this isn’t the only kind of outage we’re seeing now.
In the new paradigm, as a result of climate change we’re living with more extreme weather conditions. Drought, tree-killing diseases and pests, extreme heat, record-breaking winds — all of these have caused a sharp increase in fire danger. On the power-supply side, PG&E filed for bankruptcy based on fire liability. Now they’re shutting down their infrastructure in advance of damage, powering down hundreds of thousands of homes and businesses all at once, and for days — perhaps, they’ve hinted, weeks — at a time. These are much longer outages than our customers and Cruzio are accustomed to.
The climate situation isn’t getting better, and we don’t expect PG&E’s response to improve any time soon. Our community and our business have to adapt.
We’re Looking at Enormous Investments to Deal with Power Outages
Cruzio, as a provider of vital services, has to adapt even faster and more thoroughly than most other businesses.
So the tens of thousands of dollars per year we spend on backup power will have to double and triple. We will likely spend hundreds of thousands of dollars on power backups in the next year or so.
Our staff has been drawn thin by the new power paradigm. Over the last weekend we had techs napping in our break room as they waited for their next generator-refueling shift. We’ve added staff to build out our network, and now we’ll need to hire and train more people to take care of our equipment at our many sites. This, too, is costly: responding to power emergencies slows down our network builds. We would rather be building fiber. But until our state figures out a different way, this is the world we have to deal with.
Our Number One Priority is Always Keeping our Current Customers Up and Running
Along with making our network more resilient around the county, we’re improving our in-building power redundancy as well. When customer power is out, we welcome folks to come down to our office and use our internet. We even lower our coworking fees during outages so non-members can get the internet they need.
And we’re working on upgrades to our coworking electrical backups, so that we can accomodate more people. In a long, wide-area power outage, people need a place to go.
But We Need a Better Way
Just to touch on the larger picture: small diesel-fueled machinery like generators are bad for the environment, but people are understandably buying them up. With these long outages the small generators will almost certainly proliferate, and that will exacerbate the climate problem. We need to find other solutions. Cruzio, for example, is looking at using more solar power. We hope that our community as a whole recognizes this issue so we can all address it together and find a better way.
Seeing the 4 am status updates from Dan or spotting Colin, exhausted, between shifts makes us very grateful for the team we have and how much care they put into their work. Many thanks to them for keeping us going. And let’s hope the wind dies down soon so the power can come back on.
