What "capped by discovery budget" means

At the bottom of your crawl summary you'll see one of these:

• "2,273 of 2,273 discovered pages crawled — complete" — we found everything and crawled everything.
• "2,273 of 2,348 discovered pages crawled — capped by discovery budget" — we deliberately skipped some.
• "1,354 of 2,126 — stopped early" with a reason — something went wrong; the reason tells you what.

The middle one confuses people, so here's what it actually means.

Most sites have a huge number of pages that exist only to list other pages: tag archives, category archives, author archives, date archives, and page 47 of a blog listing. On a WordPress site with a few thousand posts, these can outnumber your real content three to one. They're near-worthless for SEO, they're generated from one template, and crawling 2,500 of them tells you nothing that crawling 150 of them doesn't.

So TaskGlaze samples them. Up to 150 terms per archive type, then it stops. It also stops paging through a listing once two pages in a row produce no new content. Your actual pages — posts, products, service pages — are never affected, because we reach those through their own URLs regardless of whether we crawled the tag archive they happen to sit under.

That's why a site with 2,348 "discovered" URLs gets 2,273 crawled and still counts as a complete picture of your content.

When it matters: if you genuinely need every tag archive audited — an ecommerce site where category pages are money pages, say — email support@taskglaze.com and we'll look at it for your project.