Custom Link Crawler
CrawlHawk's Custom Link Crawler scans any website on your instruction and returns a structured dataset of links — with full configurability over scope, link types, and output format. Use it for SEO site audits, broken-link maintenance, internal linking reviews, outbound link inventories, and any link-analysis task that demands more than a sitemap generator can provide.
What the Custom Link Crawler Does
While the XML Sitemap Generator focuses on producing a sitemap file, the Custom Link Crawler delivers raw link data optimized for analysis. For each link discovered, you receive the source URL, target URL, anchor text, HTTP status code, link type (internal, external, file, image), rel attributes (nofollow, sponsored, ugc), and crawl-depth position. The result is a downloadable dataset in CSV, JSON, Excel, HTML, or PDF format ready for spreadsheets, BI tools, or your own pipeline.
Configurable Scope
Pick the scope that matches your task: Full Domain to crawl an entire site, Main Domain Only to exclude subdomains, Subdomain Only to focus on one subdomain, Single Path to restrict to a section, or Single Page to extract from one URL only. Each scope is described in detail on its own page.
Link Types You Can Collect
Combine any of six link categories within one crawl: Internal Links (within your domain), External Links (outbound to other domains), Broken Links (4xx / 5xx errors), Orphan Pages (pages with no inbound internal link), File Links (PDFs, downloads, documents), and Image Links (every image source URL). You only pay for what you crawl; all link types are included in every plan.
Export Formats
Export to CSV (for spreadsheets), JSON (for programmatic use), Excel XLSX (for reports), HTML (for visual review), PDF (for client deliverables), or XML sitemap format (for search engines). All formats include the same dataset — choose what fits your downstream workflow.
Common Use Cases
SEO professionals use the Custom Link Crawler for site audits, technical SEO reviews, internal linking analysis, Broken Links maintenance, and content silo verification. E-commerce teams crawl competitor product structures and analyze category navigation. Content teams audit External Links for brand-safety. Data teams build link graphs for further analysis. The tool is intentionally general-purpose.
JavaScript-Heavy Sites
Some modern sites (Amazon, YouTube, LinkedIn) load content dynamically via JavaScript. The Custom Link Crawler can be configured to render JavaScript so links generated client-side are also captured. JavaScript rendering uses additional credits per URL — exact amounts are shown before crawl start.
Pricing and Free Tier
Credit packs start small. One credit per URL crawled, with JavaScript rendering and AI extraction costing extra credits per URL. The free tier covers 500 URLs without a credit card. Credits never expire — see the Pricing page.
→ Run a custom crawl · Want a sitemap output? Use the XML Sitemap Generator
Frequently Asked Questions
What's the difference between this and the XML Sitemap Generator?
The XML Sitemap Generator produces a sitemap file for search engines. The Custom Link Crawler delivers raw link data with full attributes for analysis. Both use the same underlying crawl — the difference is in what you do with the output.
Can I combine multiple link types in one crawl?
Yes. Select any combination of Internal Links, External Links, Broken Links, Orphan Pages, File Links, and Image Links. They are all reported in one consolidated dataset, and every type is included in every plan.
Does it follow robots.txt?
By default yes. CrawlHawk respects robots.txt, terms of service, and other access policies of the target website. For authorized crawling that requires overriding these (your own sites, sites with written permission), see the Acceptable Use Policy.
How are credits consumed?
One credit per URL crawled. JavaScript rendering and AI extraction modes use additional credits per URL — exact amounts are shown before crawl start. The free tier covers 500 URLs.
Can I schedule recurring crawls?
Yes. Crawls can be scheduled to run automatically at the interval you choose, or initiated manually from your account or via the API. Scheduled crawls consume credits per run in the same way as manual crawls.
Is there an API?
Yes — API access is available. Developers get a separate API documentation portal accessible from the main navigation.