A photo of our completed backup link as the sun sets in Watsonville.
Over here at Cruzio, we’re familiar with challenges. When trouble strikes, in the internet infrastructure biz, it strikes hard. We’re always prepared for any eventuality, and we do our best to make sure all of our friends and customers don’t even realize a crisis is happening at all. Case in point: a major fiber cut in Watsonville yesterday afternoon.
Working with fiber can be incredibly rewarding, as it lets us get previously unimaginable internet speeds throughout the county. That said, fiber cable is made of glass, and glass can be broken with enough force. Yesterday, a construction crew’s backhoe used a lot of force to break the 288 separate strands of fiber that form our backbone into Watsonville while taking out a water main in the process. At around noon, we saw the effects of the fiber cut suddenly pop up on our network, and our infrastructure team sprang into action.
A Week’s Worth of Work in a Matter of Hours
Within one hour, our team had developed a plan to build a new 10 gigabit connection to bring the network back online using Aviat Networks’ millimeter wave radio platform. This isn’t a quick fix, by the way, infrastructure at this level would normally take weeks of planning and at least a few full days of work to build. Our team planned it out in less than an hour and was rolling out to get it built very shortly after that. All told, we were able to complete the initial build of our backup infrastructure in around 4 hours, and we were up and running in less than 6. That’s at least 3 full days of work for a normal infrastructure team, completed in less than 8 hours. Do they have superpowers? Perhaps.
By the time the fiber was restored at around midnight, many of the people in our field operations team had worked well over 12 hours that day to make sure our customers were affected as little as possible. Now, a day later, everything is restored, and all is normal again. In fact, better: we now have a permanent backup in place to avoid even small disruptions.
So maybe today we’re a bit sleepy. But we’re proud of the quick, efficient and responsive work we did in making sure as few people as humanly possible felt the wrath of The Great Watsonville Fiber Cut of 2020. Kudos to the team who’s capable of such extraordinary work: Frost, Ali, Mark, Colin, Cam, Jay, Spencer, Hans, and Luis.
