Parse robots.txt and explain its rules in plain English.
Fetch and parse robots.txt for any domain. See user-agent blocks, allow/disallow rules, sitemaps, and a plain-English summary.
A fast technical SEO check for crawlability and sitemap discovery.
What this tool helps you do
Rule parsing
Extract user-agent, allow, and disallow rules automatically.
Sitemap discovery
List sitemap URLs declared in robots.txt.
Plain-English summary
Get a human-readable overview of the file.
How it works
Enter the domain.
Use the homepage or any page on the target domain.
Fetch and parse.
The tool reads robots.txt and breaks it into structured rules.
Review the summary.
Confirm crawlability, spot missing sitemaps, and compare user-agent blocks.
Use cases
Where teams use this tool
Technical SEO audits
Check crawl instructions and sitemap setup quickly.
Migration QA
Verify robots.txt rules after domain or platform changes.
Competitor research
See how competitors manage crawler access.
Developer-friendly
Simple Worker endpoint
/api/inspect?url=https://example.comEach free tool runs as a small Cloudflare Worker with a public HTTP interface. That makes them useful for browser use, SEO pages, and automation flows.
Open source
Use the repo, fork it, or deploy your own copy
Other free tools
Browse the Lindo tool library
Website Score
Free Website Score tool. Get mobile, desktop, and conversion scores with actionable insights for any public URL.
URL to Markdown
Free URL to Markdown tool. Convert any public page into clean markdown for AI workflows, briefs, and content reuse.
URL Screenshot
Free URL Screenshot tool. Capture desktop or full-page screenshots from any public website URL.
Meta Tag Checker
Free Meta Tag Checker. Audit title tags, meta descriptions, canonicals, Open Graph, and Twitter tags on any page.
Open Graph Preview
Free Open Graph Preview tool. Preview how your page looks when shared using Open Graph metadata.
Heading Structure Audit
Free Heading Structure Audit tool. Check missing H1s, heading order, and hierarchy issues on any page.
Sitemap Inspector
Free Sitemap Inspector. Inspect robots.txt sitemap declarations, sitemap.xml files, URL counts, and sample entries.
Content Summary Generator
Free Content Summary Generator. Summarize any public page into concise bullets and reusable notes.
Tech Stack Detector
Free Tech Stack Detector. Detect likely frameworks, CMSs, analytics scripts, and other technologies from a page.
Landing Page Scorecard
Free Landing Page Scorecard. Score any landing page for messaging, trust, CTA quality, and SEO basics.
Link Extractor
Free Link Extractor. Extract internal and external links from any public page with anchor text and absolute URLs.
Image Extractor
Free Image Extractor. List images, alt text, dimensions, and likely hero images from any public page.
Redirect Checker
Free Redirect Checker. Follow redirect chains, see status codes, hops, and the final destination URL.
Readability Score Checker
Free Readability Score Checker. Estimate Flesch Reading Ease, grade level, and reading time from any public page.
SERP Snippet Preview
Free SERP Snippet Preview tool. See how your page title and description appear in Google search results.
Keyword Density Checker
Free Keyword Density Checker. Analyze top repeated words and phrases from any public page with density percentages.
Broken Link Checker
Free Broken Link Checker. Find broken links, 404 errors, and timeouts on any public page.
Twitter Card Preview
Free Twitter Card Preview tool. Preview how your page looks when shared on Twitter/X using meta tags.
Content Word Counter
Free Content Word Counter. Count words, sentences, paragraphs, and estimate reading time from any public page.
