restrict number of concurrent scraping tasks a user can have #infinityxml

Fix movies scraper because data format changed overnight #life

Update scraper, now storing movie director, quality and still #life

restrict size of scraping task #infinityxml

improve scraping script

Add retry delay to apartment scraper because website goes down every night #life

speed up scraping script

Finish a web scrapper that make the process of adding bulk new quality replays in a matter of 3 clicks #demtovideo

refactor rescue_on calls on scrapers and add other common errors that should be retried automatically #reva

I just pushed an update that: Has 3,000 URL max per scrape; Prints to sheet in batches of 10 as it scrapes; Will continue scraping even if you close sheets, log out, etc.; Will continue scraping past the previous 6 minute timeout restriction #infinityxml

relaunched the scraping machine #thecompaniesapi

Fix rate limiting issues

add scraper service to grab video urls #watchr

wait for rate limit #therapistai

experiment with auto-retrying limit stops #rush

#slackdiscord Add failsafe on rate-limit so I know a) when to try again and b) to avoid getting timed out for eternity

Notice Watchdog is still getting blocked by X on occasion even though I have much better resiliency now with auto refresh. Investigating hijacking the requests so I can see the X-Rate-Limit-Remaining header to know when to start scraping again instead of refreshing blindly #watchdog !private

see #remoteok recover a bit after fixing the scrapers that were broken for 10 months

add rate limiting for #remoteok API to prevent abuse

fix rate limiting issue