pin nvidia driver version on #mirage so that next k8s upgrades and node pool upsizes do not create incompatibilities on our workloads due to driver auto-upgrades

finally confirm full recovery of all #mirage ai services due to NVIDIA driver update incompatible with our CUDA version

struggle to recover #mirage from a node reboot-induced outage, due to NVIDIA driver being bumped by cloud provider from version 550 to 570, thus requiring a CUDA update on our images, but our Python code is not compatible, so need to fix LOADS of things

experience scaleway general outage of gpus/k8s #mirage

rack a new #mirage ai api route to get spam score of documents

fix #mirage total downtime due to OpenAI total downtime, which hanged our HTTP pipeline and prevented our self hosted SLMs and LLM to work

automate kubernetes docker image deployments from GitLab CI for #mirage

finish reading all of kubernetes documentation for #mirage and possibly #crisp in the future

automate backups of #mirage data on kubernetes using a k8s cron job

finish migrating #mirage kubernetes intel and nvidia gpu instances to scaleway, getting last-generation NVIDIA L40S + L4 GPUs, running much smoother now! (previously: old A40 and A16)

finally ready to migrate #mirage kubernetes gpu cluster from vultr to scaleway!

mess with kubernetes, hosting a redis in k8s is really too complex #mirage

Sunday morning doing #mirage infrastructure work in hotel lobby

provision batch of L4 and L40S GPUs at Scaleway for #mirage since our account got validated and quotas lifted

experience 6 hours GPU total downtime at Vultr, knocking down all #mirage AI services, will migrate to Scaleway cause Vultr are 🤡

make some Rust async upgrades on #mirage REST API to improve scalability and fault-tolerance

deploy support for json schemas to #mirage prompt api

done migrating #mirage and #saasmakers to Nuxt 3

improve #mirage SSE API so that we can stream more than LLM output chunks and provide JSON data for new secret #crisp product

fix gated ai model this morning on #mirage which created downtime after k8s node restart since a model could not be pulled anymore from huggingface, i had to manually accept model ToS wtf