New to WIP?
We're a group of makers shipping together. We help each other stay accountable and reach our goals.
Apply for access
Topics
Back
Question
I have thousands of pages to index on Google. How to do this without ruining my domain rep?
I've heard that submitting a sitemap that goes from 0 to 10,000 pages in one shot ruins your domain rep.
Have any of you done something like this, and if so, what was your experience?
I've searched and found varying accounts, with some people claiming it's fine and just to YOLO add all of them, whereas others mention only adding a few pages per week to avoid being penalized heavily.
This is for https://atgatt.co where I need to index the "details" page for every single product in my database of which there are many thousands (there are already pages generated for all of them - dynamically at the moment on the client, may move to SSG/SSR for better SEO juice)
Have any of you done something like this, and if so, what was your experience?
I've searched and found varying accounts, with some people claiming it's fine and just to YOLO add all of them, whereas others mention only adding a few pages per week to avoid being penalized heavily.
This is for https://atgatt.co where I need to index the "details" page for every single product in my database of which there are many thousands (there are already pages generated for all of them - dynamically at the moment on the client, may move to SSG/SSR for better SEO juice)
👋 Join WIP to participate
So first, there's a way for you to create several smaller sitemaps with like 1000 URLs each then link them together by a sitemap index file. It's way easier on Google too.
Batch these sitemaps in increments, so don't just submit a shit ton of small sitemaps at once.
I'd highly recommend you start with high-priority pages first and batch those URLs together before moving on to low-priority pages. You get to define what "high-priority" means for you -- i.e. top selling, high search intent, etc.
SSG and SSR >>> CSR.
(A link if you want to know more context: medium.com/@natelapinski/how-…)
Even then, you're going to want to ruthlessly optimize every single page for crawlability (metadata, titles, descriptions, internal links, the works).
Are you on a shared server? If so, that'll impact your strategy. (If not, go fucking wild -- but not too wild lol.) Prioritize high-priority pages and use robots.txt.
Monitor your GSC for any errors. You'll want to look out for crawl errors, indexing issues, and warnings. Fix those as soon as they pop up. It's always good practice to monitor server logs too.
Before you submit the batches, test with a smaller batch (like 500 URLs) then watch to see how bots handle it. Pay attention how Google crawls and indexes them BEFORE you submit larger batches. Refine the process if needed.
There's more, but this should get you started.
I haven't worked on a website EXACTLY like yours, but I've handled large websites for clients (tens of thousands of pages), and it was a process getting everything properly indexed. Be patient, monitor the process, adjust where needed.
Thanks! Yep, already have the sitemap-index.xml + sitemap-n.xml thing working so that won't be a problem. I'm just concerned about the total number of pages even if spread across a few sitemaps.
Makes sense to do smaller tests, so I'll go with that approach.
If I’m not mistaken, irrespective of the number of pages you submit, Google will index about 30 pages per month at most.
Source: @dannypostma SEO course
I have my doubts since that would be literally 28 years of waiting best case haha. I’ll try it myself and report back with what Google does.
so, I just checked again the video where he mentions that -- and maybe the 30-40 pages was just an example he was giving, and not applicable to all sites. What he is referring to is ultimately that every site has a "crawl budget", and google limit how many pages it indexes, depending on the site rank/authority.
"Google limits the number of pages it will crawl and index from your website each month. If you dump a thousand pages all at once, Google might only index a small percentage, prioritizing the most critical. Spread your content publishing over time to avoid being perceived as a spammer. This strategy ensures your most important content gets indexed first."
Got it, thanks for this insight 👍