Back
Similar todos
Load previous page…
Rewrote scraper. Second attempt is always significantly better than the first one.
watched web scraping node js tuts to improve my crawlers
worked on how to test/keep a scraping project updated more easily
wrote reddit crawler #dvlpht
getting crawled at weegee.ch - by aggressive crawlers which brought the site down. Gonna do the simplest thing by blocking the IPs. As a crawler myself I know that this usually causes enough troubles to stop a crawler.
rewrote one of the running website scrapers to use playwright so that the JS bits get rendered
changed crawler so i can keep css from workdays 
#sportstechjobs
🐍 worked on a quick reusable web spider since everything else is overkill
get crawler working with html based structured data #joopie
create worker pool to improve performence for crawlers #joopie
reimplement core crawler in nodejs and puppetteer for javascript rendering #joopie
fixed crawler bug to generate slugs #sportstechjobs
finished out my day updating scraper-proxy to support css and xpath selectors along with a few features I haven't seen anyone do yet.
added two circles to #sportstechjobs crawler
worked on job website tests on one of the scrapers
double check code and create task for reedit crawler #dvlpht
fixed a bug in one of my job scrapers
write crawler who checks if there is ANY link on a given directory which is do-follow - already corrected some mistakes in my guide with the help of it 
#backlio
re-worked one of my job scrapers and trying to decide if I should keep it or keep moving forward on it