relaunched the scraping machine #thecompaniesapi

fix job that someone doesn’t want on site gets reimported after deleting, set -24h ago max timeout for sourcing jobs on Landing Jobs API scraper #remoteok

fix problem with scrapers #reva

fixed bug in cron crawler job #sportstechjobs

fix bug where scraper wouldn't skip if no job was found #plumberjobs

add request timeout #hour

I just pushed an update that: Has 3,000 URL max per scrape; Prints to sheet in batches of 10 as it scrapes; Will continue scraping even if you close sheets, log out, etc.; Will continue scraping past the previous 6 minute timeout restriction #infinityxml

check why scrapping stopped to wprk #capgo

set Google remote jobs scraper to hourly instead of every 5 minutes and save 12x less requests and money #remoteok

improve scraping script

speed up scraping script

make Twitter #remoteok scraper work again

Investigate and deploy a new try to fix vrbo and expedia more recent scraping failures #reva

Figuring out backup scrapers...

Setup an event on EventBridge to run the scraper lambda every night #domainwatchman