This seems like an elegant compromise: andycroll.com/ruby/stop-robot…

It tracks 404's, except those generated by crawlers that might crawl outdated links.