Learned about self-hosting a crawling engine today #learning There aren't too many options, which are standalone as a core package. Most are tied to some kind of application & database.