recover from an evening of server hell & murphys law at #crisp due to a digitalocean backup slowing a mongodb cluster server so much that it led to full cluster collapse and 30m #crisp downtime

Rebuild Hetzner server with backup from last night because I made a 'lil mistake 😅 #nichewit

fixing mongodb error #nbpai

confirm #photoai on Hetzner works but it has no database yet so it dies there

done with mongodb migration to a dedicated server

contact Hetzner because of random reboots during the night #spectate

discuss with Hetzner about server #rewardsflights

omg 8 hours dealing with mongodb #freelancegig

enable backups on #photoai Hetzner box

Still having network issues at Hetzner, start planning migration once I can get access to data again

Try upgrading the RAM on a Hetzner dedicated server to bridge the gap until migration #freiheitswolke

🛠 repair broken raid on #berty server

make dedicated 2 vCPU Hetzner box for #photoai

fixed mongo db bug #sportstechjobs

Get a Hetzner instance hacked because I was doing tests with Kamal last night, spawned a few test instances, probably didn't harden them well enough (because Kamal claims to leave your instances prod ready but it absolutely doesn't) so at 7am someone gained access to the instance and started abusing it #life

Got man-handled by Hetzner again

run weekend balancing of #crisp message storage mongodb cluster jumbo chunks (ie. #crisp websites that are soooo big that they cannot be moved automatically to shards w/ more disk capacity across our cluster)

Setting up a dedicated server on Hetzner

restart mongodb because disk out of space #beachnearby

Analyze stuff that can be built on new Hetzner Nvidia 6000 GPU servers: www.hetzner.com/dedicated-roo…