adding elasticsearch as a source to grafana #thecompaniesapi #saasmakers

improve elasticsearch management #thecompaniesapi #saasmakers

set up elasticsearch and kibana in devcontainer #thecompaniesapi

setting up new production elasticsearch cluster in local #thecompaniesapi

#thecompaniesapi continue digging in elasticsearch; find another root cause of one of our recent issues

#thecompaniesapi learn more about elasticsearch ; finally found what caused our migration script to end up crashing, migration should be fully completed in 15h now that its stable

#thecompaniesapi take another approach for migration & vectorization of our millions of indexed documents; ES asks for a rescroll of the whole documents to restart from similar point; should be able to do it all in 1 pass now

#thecompaniesapi quick fix on the sync script, everything goes well, feels great to see the ES cluster full green & handling the migration nicely, we should be able host up to 1B documents with that cluster! also added a validation check to our prompt to segmentation that should literally kill all possible hallucinations from our search input :))

#thecompaniesapi launch full data migration between our existing elasticsearch single-node to our fresh new 3 nodes cluster ; last major milestone before releasing 😪

#thecompaniesapi tons of improvements on the bot; prepare for the big migrations to new ES cluster; refactor of sync pipeline :))) crunching

#thecompaniesapi finally deploy the new ES cluster ; all green 2TB free storage with 3 nodes :))) ready for the next 10K+ users and 200M+ companies

#thecompaniesapi headache with elasticsearch cluster

#thecompaniesapi huge progress on elasticsearch new cluster & bot improvements :))) almost starting the data migration prior to big release!

#thecompaniesapi work on >10k lists scroll with point in time & search after from ElasticSearch

removing unused es indices #thecompaniesapi

improving elastic search mappings #thecompaniesapi

#thecompaniesapi sorting working on all columns; wrote my first painless elasticsearch scripts ; next step is very exciting 😳

#thecompaniesapi close some blocking issues ; explore new filtering type for elasticsearch ; having fun imagining all upcoming possibilities :)

#thecompaniesapi fine tuning elasticsearch index mappings ; preparing for a big reindexing of our whole db before v2

Plan how to migrate data off elasticsearch to tinybird to save $7k/yr #thieve