Notice Watchdog is still getting blocked by X on occasion even though I have much better resiliency now with auto refresh. Investigating hijacking the requests so I can see the X-Rate-Limit-Remaining header to know when to start scraping again instead of refreshing blindly #watchdog !private

see #remoteok recover a bit after fixing the scrapers that were broken for 10 months

add rate limiting for #remoteok API to prevent abuse

fix rate limiting issue

upgrade ScrapingBee API plan because #remoteok was hitting limits

#hw add rate limiting and request slowdown to protect from missuse

#madeit add rate limiting and request slowdown to protect from missuse

add rate limiting in APIs #hapramp

increase workers to process scrapers faster #reva

continue work on scrapers #pagesonpages

fix bug where scraper wouldn't skip if no job was found #plumberjobs

add rate limiting to the backend #accountableblogging

Add rate limiting to submission api #openformstack

check why scrapping stopped to wprk #capgo