As climate change escalates, several things become clear to me:
- The only thing capitalists are consistently competent at is fucking over their workers
- The industry doesn’t have the impulse control to slow its own demise
Most of the consequences of this are horrific, widespread, and harrowing to consider. This one’s a little different. I wanted to get some opinions from some devops folks who are more knowledgeable about the specifics of their work than I am.
Here’s the basic pitch. As climate change accelerates, freak weather patterns which span continents will become more common, as will infrastructure failure. We know that a key aspect of reliable offsite backups is to have proper separation of dependent systems. So if you think your servers are completely “redundant” because they run on separate VLANs and electrical circuits, but are housed in the same data center, sucks for you when the power goes out for longer than your generators can last. This is a major reason we have offsite backups. Trillions of dollars of data is stored on magnetic tapes all over the world (is this still the standard medium? Idk). But as catastrophic weather and infrastructure failure becomes more common, these separated systems become entangled again. If your offsite backups are on the east coast and your data center is on the west coast, having one building burn down in a wildfire while the other experiences rolling blackouts is not a great situation to have to prepare for.
So I’m curious how behind the curve I am on this, whether there are people at cons talking about this, etc. I assume the big players have contingency plans for this sort of thing, but a lot of companies don’t. Just another example of how capital can’t help but create the conditions of its collapse.
Many many years ago I worked doing sysadmin for a large pharmaceuticals company. This was pre-cloud era, so they had (and probably still have) a large data center for their Europe/Middle East/Africa operations. One day, hundreds of servers went offline as power distribution for dozens of racks went out. They all had uninterruptible power supplies, and there was a duplicate power distribution path in case one went out, with separate mains supply. The whole 9 yards.
Except someone had effecfively plugged everything into the same power distribution path twice.
Hundreds of thousands spent on that redundancy and someone basically fucked it by plugging it in wrong :data-laughing:
I’m pretty sure most large enterprises are closer to utter disaster than they realize, they just haven’t been actually stress tested outside of known scenarios.
This is my intuition, especially as things trigger domino effects. I could definitely see an IT equivalent of the supply chain failure happening in the next decade. Not the exact same mechanisms of course, but just things being more interdependent than was expected and no one keeping an eye on the system as a whole
:100-com: as someone that now works at godforsaken startups, it’s a fucking miracle any of these apps work, even as badly as they do. Years upon years of people taking shortcuts to get promotions, vest their equity and leave to move onto the next thing. Rinse and repeat. There’s no resilience or flexibility in many codebases more than a couple of years old. At my current company that has hundreds of engineers, there’s one or two people who approximately understand how our core business logic works.
What’s fun is that if the gears of privatisation keep turning, more and more parts of the state will be entrusted to these Rube Goldberg machines of careerism. Especially when Artificial Intelligence woo is sold as a means of addressing the many vectors of collapse. I honestly hope there’s a widespread technological disaster before too long, from that perspective.
Several datacenters here in the EU do regular blackout-tests, where they actually switch a power feed to the backup generators, in order to verify that things actually fail over like they should.