Scanning & CrawlingDocumentation
Page Collection (Crawling)
Use the crawler to discover pages and build your coverage map.
What the Crawler Does
The crawler navigates your site and discovers internal links to build a page inventory.
Discovered pages are stored under your project and can be scanned immediately.
Start a Crawl
- Go to your project and open the Runs tab.
- Select Page Collection and start the run.
- Monitor progress as pages are discovered.
What You Get
- A list of URLs with titles and status metadata.
- A run entry with status, timing, and totals.
- Optional sitemap files for navigation review.
Important
Dynamic or gated pages
Pages behind auth or blocked by robots may not be discoverable in a crawl. Add those pages manually or via sitemap upload.