Sometime early this morning, our database started having issues, most likely due to a disk corruption issue.
Unfortunately the postgres database was unrecoverable, and we had to restore from a nightly backup, so a few hours of posts and comments will be missing.
Apologies again for the downtime, and for the disruption.
Do you know that you are doing to me?! I actually had to work while lemmy was down...
How does lemmy federation work in this case? Conceivably after being restored from backup the lemmy.ml instance could see those few hours of lost history as federated to other lemmy instances and resync it back as the host instance. Obv I'm vastly oversimplifiying things but what happens today?
I believe they're lost, as recieved apub content doesn't have the original post / comments, but I'm not totally sure. cc @nutomic@lemmy.ml
That doesnt work because we generally cant trust remote servers. Plus we dont even know where to fetch from, so wed have to run a complete crawl of all known instances which isnt practical.
No worries! Is there an announcement anywhere in case it's down? Mastodon? Matrix?
We posted in some of the lemmy matrix chats, but ya we should also use our mastodon too.
it'd be helpful to post to sth like masto yeah, as its much more public facing than matrix.
I AM TALKING IN ALL CAPS BECAUSE I AM MAD
HOW DARE THIS FREE SERVICE I USE GO DOWN HOW EVER WILL I POST MEMES FROM MY OPAL THRONE
Thanks for coming to my Ted talk.
also thank you for hosting us and working on the issue!I saw the "502 Gateway" HTTP error message, but I never even closed my browser tab. I refreshed again today and Lemmy.ml was back online! Thanks for your hard work!
Please tell me you've been saving this picture just for this type of occasion. 😂
Idk what Lemmy.ml is is it some sort of liberal instance that refuses to engage with chapochat
lemmy.ml is a lemmy instance run by the developers of lemmy. lemmy.ml is federated with hexbear, so you can read their posts and we can read your posts.
I hope none of you have blocked my community because would make me angry
the owner, dessalines, is a massive comrade and writes fantastic essays. you should check out his stuff
It's the developer instance. It has no inherent or declared political leaning, though in practice pretty leftist and the devs are communists so.
