Troubleshoot Hetzner storagebox maintenance crashing MongoDB server causing downtime #retently

recover from an evening of server hell & murphys law at #crisp due to a digitalocean backup slowing a mongodb cluster server so much that it led to full cluster collapse and 30m #crisp downtime

Rebuild Hetzner server with backup from last night because I made a 'lil mistake 😅 #nichewit

Bought a new hetzner server to move things off of vercel and expensive cloud db.

fixing mongodb error #nbpai

confirm #photoai on Hetzner works but it has no database yet so it dies there

done with mongodb migration to a dedicated server

contact Hetzner because of random reboots during the night #spectate

discuss with Hetzner about server #rewardsflights

People need to run servers on Hetzner in HA mode, cause they do go offline for various reasons, especially important of your app needs to be up all the time

omg 8 hours dealing with mongodb #freelancegig

enable backups on #photoai Hetzner box

Still having network issues at Hetzner, start planning migration once I can get access to data again

Try upgrading the RAM on a Hetzner dedicated server to bridge the gap until migration #freiheitswolke

🛠 repair broken raid on #berty server

make dedicated 2 vCPU Hetzner box for #photoai

fixed mongo db bug #sportstechjobs

Get a Hetzner instance hacked because I was doing tests with Kamal last night, spawned a few test instances, probably didn't harden them well enough (because Kamal claims to leave your instances prod ready but it absolutely doesn't) so at 7am someone gained access to the instance and started abusing it #life

Got man-handled by Hetzner again

run weekend balancing of #crisp message storage mongodb cluster jumbo chunks (ie. #crisp websites that are soooo big that they cannot be moved automatically to shards w/ more disk capacity across our cluster)