Hey folks!

Unfortunately, roughly 2 hours ago, lemm.ee went offline. The cause was our load balancer: it suddenly decided that all of our servers had become unhealthy, despite all health checks responding successfully when I requested them directly. In such cases, the load balancer stops serving all requests, effectively meaning that lemm.ee is unreachable for all users. I am still not sure what exactly caused the issue, but I will try to investigate more over the weekend.

For now, we have partially recovered, and I am continuing to work on remaining issues. Hopefully we will be back to 100% very soon. Sorry for the inconvenience!

  • fossphi@lemm.ee
    ·
    4 months ago

    Thanks for the quick fix! What did you have to do to get the load balancer working again?

    • sunaurus@lemm.eeM
      ·
      4 months ago

      For now, I just redeployed all of our servers completely, but as I don't know the actual root cause of the issue yet, I'm still investigating to figure out if anything more is needed.