Free Tool
Robots.txt Checker
Review your robots.txt crawl rules, sitemap hints, and common SEO/security mistakes before they affect discovery.
Sign in with Google or GitHub to run the scan. Start with a free scan.
What this tool checks
This page uses ZeriFlow's deterministic website security engine and focuses the guidance on the configuration area above. For the full report, run a complete free security scan.
Key Checks
Crawl Directives
Reviews User-agent, Allow, and Disallow rules for broad blocks or accidental crawler restrictions.
Sitemap Discovery
Checks whether your robots.txt points crawlers to a valid sitemap location.
Sensitive Path Signals
Highlights cases where robots.txt publicly advertises private-looking admin or backup paths.
Syntax Clarity
Looks for confusing or conflicting rules that can behave differently across crawlers.
Recommended Baseline
Location
Serve robots.txt from the domain root, for example https://example.com/robots.txt.
Sitemap
Include a Sitemap directive pointing to your XML sitemap.
Privacy
Do not list secret or sensitive paths as a security control.
Testing
Review rules after every major CMS, framework, or deployment change.
FAQ
What does robots.txt do?
robots.txt gives search engine crawlers instructions about which paths they may crawl. It is a crawl hint, not an access control mechanism.
Can robots.txt protect private pages?
No. robots.txt is public and should never be used to protect sensitive content. Use authentication and server-side authorization for private pages.
Should robots.txt include a sitemap?
Yes, adding a Sitemap directive helps crawlers discover your XML sitemap location, especially when it is not at the default /sitemap.xml path.
What is a bad robots.txt mistake?
Accidentally blocking the whole site with Disallow: / for important user agents can remove pages from search discovery over time.
Need the full security picture?
ZeriFlow combines deterministic website checks across headers, TLS, DNS, cookies, and email security with monitoring, reporting, and AI-powered developer workflows where implemented.