Back
Similar todos
make stats page to track if the robots are doing their job 
#hotellist
researching robots 
#mylife
fixed robots.txt 
#sportstechjobs
try to extract robotics data 
#cappr
robots.txt 
#melies
update robots blocking lists
add robots.txt 
#blip
add robots.txt 
#freelancegig
add some stats so I can see if the robots working or being lazy #hotellist
fixed robots.txt as well 
#rc
add robots.txt #cronheatmap
add robots.txt file 
#changeloghost
#dw add robots.txt (so that the robots find something to read)
see robot stats since 12 hours ago #hotellist
