Back
Similar todos
Restart movies scraping task that failed last night due to excessive timeouts #life
Fix movies scraper because it broke overnight #life
restrict number of concurrent scraping tasks a user can have #infinityxml
Fix movies scraper because data format changed overnight #life
Update scraper, now storing movie director, quality and still #life
restrict size of scraping task #infinityxml
Finish a web scrapper that make the process of adding bulk new quality replays in a matter of 3 clicks #demtovideo
I just pushed an update that: Has 3,000 URL max per scrape; Prints to sheet in batches of 10 as it scrapes; Will continue scraping even if you close sheets, log out, etc.; Will continue scraping past the previous 6 minute timeout restriction #infinityxml
relaunched the scraping machine #thecompaniesapi
Fix rate limiting issues
add scraper service to grab video urls #watchr
wait for rate limit #therapistai
#slackdiscord Add failsafe on rate-limit so I know a) when to try again and b) to avoid getting timed out for eternity
Notice Watchdog is still getting blocked by X on occasion even though I have much better resiliency now with auto refresh. Investigating hijacking the requests so I can see the X-Rate-Limit-Remaining header to know when to start scraping again instead of refreshing blindly #watchdog !private