Back
Similar todos
Load previous page…
✅ shipped a robots.txt file to ignore /cdn-cgi/ 
#lfk
add static robots.txt file 
#scoredetect
Fix Reddit bot that was returning 403 because missing User-Agent 
#jobboardsearch
block tor exit nodes from accessing 
#crisp by challenging them, only humans-over-tor shall pass
"block user" on web 
#unistore
#ws add robots.txt
#ws add robots.txt
block requests from kproxy
#dw add robots.txt (so that the robots find something to read)
fixed robots.txt as well 
#rc
block blacklisted IPs and mturk referer 
#wip
handle being blocked by user 
#sendmelater
remove Analytics for if robot is testing 
#remoteok (for @daniellockyer)