Website Mapper
Discover all URLs from a website using intelligent sitemap analysis and crawling. Returns a list of all accessible pages.
View detailsInputs
Loading workflow structure...
Overview
Website Mapper discovers the URLs on a website so you can understand its structure before scraping, auditing, or planning content. Use it when you need a map of accessible pages, not the full page text, and want to choose the best URLs for follow-up research.
Use cases
- Find product, pricing, blog, and resource URLs before a competitor audit.
- Build a source list for later scraping or content research.
- Check how a site is organized across root pages and subdomains.
Input tips
- Enter the root website or section you want to map.
- Use sitemap controls to prefer, include, or skip sitemap discovery.
- Set a limit when you only need a manageable sample of URLs.
- Use search when you want URLs related to a specific topic or keyword.
- Leave query-parameter deduplication on unless URL variants matter.
Expected output
The AI Tool returns the normalized root URL, total discovered link count, and a list of discovered URLs with page titles or descriptions when available.
Caveats
- It maps URLs; use Web Page Scraper when you need page text, media, or screenshots.
- Large sites can return many URLs, so apply limits for focused research.
- Discovery depends on public access, sitemap quality, crawlability, and site structure.
- Some metadata fields can be missing even when URLs are found.
Related AI Tools
Web Page Scraper
Extract clean markdown content from web pages. Supports single URLs or batch scraping of multiple URLs.
Exa Search
Use Exa Search for citation-backed query-first web research, company and people discovery, news, PDFs, research papers, personal sites, financial reports, concise highlights, summaries, and deep synthesized search. If you already have URLs, use Exa Contents instead.
Exa Contents
Use Exa Contents to retrieve clean page content, highlights, summaries, links, images, subpages, and per-target statuses from known URLs. Use Exa Search instead when you need to discover pages from a query first.