The biggest lesson from scraping was that I should separate the discovery process from scraping and that I should have a data retrieval method routing and a separate service for each source.