Common Workflows

Standard ways to use Crawlith in your daily audit and monitoring tasks.

Quick Structural Audit

Run a fast scan of the top levels of your site to assess general health:

crawlith crawl https://example.com --limit 200 --depth 3

Goals: Catch 404s, discover missing titles/meta, and view the high-level Health Score.

Identify your most authoritative pages and find internal link leaks:

crawlith crawl https://example.com --compute-pagerank --orphans --sitemap

Goals: Map authority flow with PageRank, find Orphaned Pages, and ensure Sitemap coverage.

Compare your current site structure against a previous baseline:

crawlith crawl https://example.com --compare ./reports/baseline/graph.json ./reports/current/graph.json

Goals: Spot newly added or removed URLs and track structural changes over time.

Integrate Crawlith into your build pipeline to prevent breaking SEO quality:

crawlith crawl https://example.com --fail-on-critical --limit 500

Goals: Automatically fail a build if Critical Issues like broken links or redirect loops are detected.

Spot repeating content layouts or near-duplicate sections:

crawlith crawl https://example.com --clustering --cluster-threshold 10

Goals: Group structurally similar pages into Clusters to audit boilerplate density.