Improve movies scraper to only scrape priority countries #life

Restart movies scraping task that failed last night due to excessive timeouts #life

Fix movies scraper after seeing some un-expected data #life

Fix movies scraper because it broke overnight #life

restrict number of concurrent scraping tasks a user can have #infinityxml

Fix movies scraper because data format changed overnight #life

improve scraping script

speed up scraping script

refactor rescue_on calls on scrapers and add other common errors that should be retried automatically #reva

Swap IPs and finish scraping all data necessary to relaunch #atgatt because they’ve added too much bot protection

I just pushed an update that: Has 3,000 URL max per scrape; Prints to sheet in batches of 10 as it scrapes; Will continue scraping even if you close sheets, log out, etc.; Will continue scraping past the previous 6 minute timeout restriction #infinityxml

relaunched the scraping machine #thecompaniesapi

Notice Watchdog is still getting blocked by X on occasion even though I have much better resiliency now with auto refresh. Investigating hijacking the requests so I can see the X-Rate-Limit-Remaining header to know when to start scraping again instead of refreshing blindly #watchdog !private

increase workers to process scrapers faster #reva

check why scrapping stopped to wprk #capgo

fix failing scrapers #reva

fix problems with scrapers #reva

put manually enqueued scrapers in a higher priority queue cause onboarding of new users too slow if we've ran the automatic scrapers and the queue was full #reva

looking into some new scraping issues

build more scrapers #pagesonpages