My job is literally unraveling shit like twitter, trying to keep the lights on with minimal knowledge
The complexity of twitter and the degree of layoffs they're rumored to have will be extremely difficult to come back from without experiencing some sort of catastrophic failure first. Will require an absolute assload of people to fix it, more than 1x the people who were fired. Maybe 2x. And they'll have to know what they're doing. Remains to be seen if those catastrophic failures will be unrecoverable
It just takes one missed maintenance task to bring everything to a pile of rubble, and my understanding is that not only did they fire everyone who does the maintenance tasks, they fired everyone who knows who was supposed to do the maintenance tasks
I think all the people talking about programmers thinking they're unreplaceable need to learn what ops/infra/SRE/security teams do and what happens when you get rid of them
Thank you. Working with a code stack you've not seen before is like learning a new language, and if the language of documentation (if indeed, there is any documentation) isn't your native language, then it's gonna be rosetta stone levels of confusion.
It's not operating normally - it's perceived to be operating normally, but it's a ticking time bomb for catastrophic failure and they laid off all the firefighters
It's also absolutely hemorrhaging money and losing revenue, advertisers are fleeing the platform and a lot of standing business deals have been cancelled because there's no one left arranging them. It's not taking down videos with DMCA claims because no one is left to deal with them, which is going to get it into legal trouble. The government is mad bc they fired the CIA regime change team. Capital might not care about long-term technical stability, but it does care about short term profits, intellectual property, and the ability to enforce its hegemony
In no way will this be looked at as a "success"
I think the next major outage will be scrutinized heavily for sure. And I have a hunch the next outage will be in the next couple months — globally-observed events with spikes in traffic that have second-level specificity tend to put huge strain on very complex systems like most modern social media sites.
Teams of people spend weeks doing capacity planning in anticipation of events like the World Cup or New Year's typically. World Cup specifically is a notorious SRE nightmare for big social media sites — you'll have people from all over the world posting at the exact same time (like, to the second/minute) when exciting things like goals happen and they'll be posting photos and video clips and excitedly spamming a million tweets.
The #1 most common cause of major site outages is increased load by far. You've got a complex system with a million little tiny gears, and if one gets overwhelmed and starts to slow down, or if a disk or something fills up, or too many things are connected to a database, or whatever then the whole thing catches on fire in spectacular ways
having everyone in the world tweeting about an incredible save, or a shitty call, or a ridiculous goal all at the same time means the traffic is super concentrated and super high. A lot of work has to go into preparing to keep things online and actively putting out fires while the events are ongoing. I've heard engineers from Instagram talk about how they always have a miserable New Years bc a lot of things always break with the increased load. And it's just not possible or practical to anticipate every potential failure mode.
Long ramble, but i'd expect it to be a slow collapse until it isn't. It can't stay online with a skeleton crew forever
that is, definitionally, not institutional knowledge. maybe they can keep the wheels from falling off for a while by finding cheap, disposable labor, but that's only creating exponentially more problems. like i understand and agree with the vague sentiment that computer jobs are all made up bullshit, but that's because their products are useless/evil, not because labor is somehow unnecessary to production or because they're not legitimately professional.
It's also true that a lot of the ancillary labor that goes into these companies is bullshit in someway. Like every boss having a secretary to berate, or the marketing department. They do work, but it's of no discernable benefit to society. Software engineering is an intellectual labor, and a potentially very valuable one at that. But it's best value is for building the kind of infrastructure that would destroy capitalism, so instead we do stupid shit with this labor.
Ah of course you are right, unless, of course, you turn out to be wrong
The employees there are being forced to work 60+ hours a week now
And it’s been less than a month
Maybe, but don't forget that whatever geniuses are desperate enough to take that job will be following the orders of a man who doesn't know what the fuck he's doing.
For example, Space X has some good engineers on their team, I'm sure, but the shuttles still leaked piss because Musk is the one calling the shots in the end.
As long as capitalism gives unearned leadership power to the rich and stupid, there is nowhere for these places to go but down, eventually.
They also fired workers in the third world though who are paid a fraction of first world workers.
I'm not sure what it is that Musk wants. Sure, they can hire cheap outsourced third world labour for some of the stuff but that still requires extensive training.
yeah i've been low-key waiting for the gravy train to stop. it's not that coders are overpaid, but they're not being squeezed hard by porky yet, and i'm sure they're dreaming of it constantly.
i dunno shit about shit with tech stuff and servers and whatnot, and i dunno the actual chances of this turning into, say, a 3 day outage at some point - but i imagine it might? seems like even that would be enough for a lot of people to consider it the end of the road for twitter and desert in droves to other social media to go yell at each other about ableist chili or whatever on. and even if they got it up and running reliably again after that it kinda seems like the damage to the site and its engagement would be basically irreparable
I have worked as an SRE, and I think you're being just a little impatient. If the previous engineers were good, it will take a while for problems to build to the point that something really breaks. Give it like 3 months.
Yeah, maybe OP turns out to be right, but it's far too soon to tell.
Twitter suddenly exploding would be too interesting and or funny which is why it won't happen
