Analyze any site's robots.txt for SEO and crawlability.
robots.txt is a plain-text file at the root of every website that tells search engine crawlers which pages and sections they are allowed (or not allowed) to access. It's the first thing Googlebot, Bingbot, and other bots look for when they visit a domain.
Common directives include User-agent (which crawler the rules apply to), Allow (explicitly permit a path), Disallow (block a path), and Sitemap (point to your XML sitemap).
Why check it? A misconfigured robots.txt can accidentally block important pages from being indexed — tanking your SEO. It can also leak admin URLs or fail to declare a sitemap. This tool fetches the live robots.txt, parses all directives, and flags common issues so you can fix them fast.
It fetches and parses the robots.txt file for any domain, showing which URLs are allowed or disallowed for search engine crawlers.
It controls which pages search engines can crawl and index. A misconfigured robots.txt can accidentally block important pages from appearing in search results.
If no robots.txt exists, search engine crawlers assume they are allowed to crawl all pages on your site.
The robots.txt file at your domain root tells search engine crawlers which paths they may fetch. This tool retrieves and parses your live robots.txt, highlights syntax issues, lists allow/disallow rules, and shows whether a sitemap URL is declared. Misconfigured robots.txt can accidentally block your entire site from Google indexing.
User-agent: * rules apply to all crawlers unless a more specific agent block exists. Disallow: / blocks everything — catastrophic if unintentional on production. Allow rules can override disallow for specific paths (Google supports this). Sitemap: directive helps search engines discover your sitemap URL — should point to https://www.yourdomain.com/sitemap.xml.
Remember: robots.txt is a polite request, not access control. Sensitive directories need authentication, not just disallow rules.
/ on production after copying staging robots.txtDisallow: /blog vs intended /blogs)A Shopify migration left Disallow: / in robots.txt from the pre-launch password page. Organic traffic dropped 90% over two weeks. The robots checker would have flagged the blanket disallow on day one.
We fetch robots.txt from the live domain once. CDN-cached or geo-variant responses may differ. We parse standard directives — not proprietary crawler extensions. For indexability beyond robots.txt, use Google Search Console.
Pre-Launch Audit Guide · Subdomain Finder · Website Health Check · View our sitemap