:crab-party: :crab-party: :crab-party:
This is the result of critical infrastructure privatization. The people's internet services would never go down!
It's so funny to me that every time this happens it turns out Amazon hosts like 80% of its own services on us-east-1 and it's just us-east-1 that's down. Why they have not diversified their own infrastructure to even just us-east-2 is beyond me.
It's probably diversified, there's just a single point of failure that's difficult to get rid of, probably a network load balancer or something.
But like, why not move that outside of us-east-1? That's literally always the one that goes down. Just move that single point of failure outside of the most used region by a large margin on AWS. Seems like a basic reliability engineering practice.
Each site has a network load balancer, I mean if it was an AWS wide one, it probably wouldn't be a single region that goes down. And again, this is just my guess. Having a hardware load balancer is how the place I worked at managed traffic between several server rooms, and it was usually the main culprit for downtime. It's a very handy device so your network doesn't get overwhelmed by traffic spikes, but afaik really hard to make redundant.
Oh I see what you mean. Yeah I guess it would make sense that us-east-1 goes down often since it's the most trafficked and it does have to have physical hardware that can fail.
Why do they get a choice in a specific server? Sure they could specify us-east, but why let them pick us-east-1 specifically?
That sort of thing already exists, as availability zones. Things like EC2 instances live in us-east-1a, us-east-1b, etc, which are comprised of separate data centers. In theory that should provide resilience to even a large-scale outage, but evidently that's not foolproof.
The reason why they let you be very precise with how you provision servers is because some applications require that servers be physically close together, especially high bandwidth stuff.
In particular, the control plane for Route53 (their DNS product) lives entirely in us-east-1, so the blast radius of a bad outage in that region is enormous.
Taking AWS down in solidarity with the striking Kellogs workers is pretty lit, ngl.
Hell yes :sicko-yes: we really need a militant software operator's union
Programmers really do hold a lot of power compared to the bosses that pay them. They just need some solidarity and they're good.
Oh yeah, at my last job one time during a casual after work chat like a half dozen of my coworkers and I were just talking about what a hypothetical "disgruntled employee" could do to cripple the company if they wanted. Some of my favorites:
- delete every IAM policy, role, and user so no one could log in to AWS and nothing internally could talk to anything else, but the company would still rack up their usual hosting bill
- write a bunch of realistic junk data to the database, scramble all the foreign keys, touch every record, delete the backups
- sell the company's domains
I mean, shit, I don't know a single devops engineer who hasn't thought about the myriad of ways they could literally shut all operations down in a way that would take their coworkers weeks to recover and would be impossible for anyone without working knowledge of the systems to fix.
Amazon warehouses all over NA are down, they usually just make everyone sit on their thumbs for several hours until it's resolved.
Glad the warehouse workers are finally getting a break :sicko-wholesome:
When I worked at one it happened a couple times, was like Christmas vacation.
:deeper-sadness: For once, my role on my current project doesn't require AWS, or I'd be celebrating with you.
Libre/Open-source self-hosting alternatives gang RISE UP! THIS IS WHY A FEDERATED INTERNET IS COOL AND GOOD
:praise-it:
You're not wrong. I'm a street Samurai not a decker. I know very little.
I think paid hosting is more because it's more cost-effective, simple, and most importantly allows companies to have servers around the world without having to own internet-connected property everywhere in the world.
Amazon web services, Amazon's cash cow — they host like 1/3 of the internet lol
Yeah, their status page has been green but a lot of their services are down so depending on how your calls are wired up it's v possible
Pretty sure that status page will be green well after the extinction of humanity
Glad you're getting some respite from the onslaught of angry and confused Karens :mao-clap:
I had some anarchist friends who used to work there, they had some real funny horror stories. They quit around the time the pandemic began, hope they are doing alright.
Ive been on call through day-long outages in us-east-1 but I'm impressed with how much we've discovered is broken vs what they say is having issues lol. Feel like they haven't acked like half the problems they have right now
That checks out and also sounds extremely unhealthy and counterproductive to fixing and preventing outages. Capitalist efficiency strikes again!
I've heard that the CEO of AWS has to personally approve all red-x statuses for core services like EC2 and S3, which also sounds counterproductive and toxic as fuck lol
When your ideology is definitely conducive to efficient operations.
