Back
Similar todos
Load previous page…
update robots blocking lists
make search engine available only for users 
#yelmair
bug: disable authorized origins while it's not used 
#apercite
per domain robots block on new system #host
add robots.txt file 
#changeloghost
#td add robots.txt
Handle invalid URLs when fetching user sites 
#mugshotbot
Prevent search engines from indexing #airbot
add a few more keywords to disallowed subdomains #changeloghost
fixed robots.txt 
#sportstechjobs
Add user agent to job offers API call as is mandatory for a new feed 
#jobboardsearch
Remove temporary notice from client website 
#freelance
added Disallow: /brands-in-sportstech/ to robots.tect because google is indexing non existing urls #sportstechjobs
bypass adblockers for 
#remotelist
✅ shipped a robots.txt file to ignore /cdn-cgi/ 
#lfk
add static robots.txt file 
#scoredetect