Sign in Join WIP
Other questions
Universal bucket in Google Cloud Platform for content, which would determine the user's location and serve content from the closest server? 1 answer
no-code tool that allows permissions? 4 answers
Not a question, just some feedback from a baby-maker to other baby-makers 5 answers
Is there a way to move the user's AppStore subscription renewal to the day in the future? Be the first to answer
Asked

Easiest way to scrape a site behind cloudflare?


Ernst Mulders
Ashish Kumar (Ashfame) 🐲
Decentralization evangelist, Entrepreneur, Life hacker, Backend engineer. Cyberpunk @ Woodpckr.com | Building AirGapped Crypto Vault live on stream

Go slow 😛

Reply
Ashish Kumar (Ashfame) 🐲
Decentralization evangelist, Entrepreneur, Life hacker, Backend engineer. Cyberpunk @ Woodpckr.com | Building AirGapped Crypto Vault live on stream

Or scrape from several proxies, but still slow. More proxies you can use, more fast you can go.

Reply
Deniz

Yeah, proxy servers helped me a lot in the past.
There are back connect proxy services out there that autorotate proxy server for every request/session.

Depends on what you are scraping, make sure you have some sort of a queue system, that will retry when scraping failed.

Reply

👋 Join WIP to participate