Robots.txt File

A robots.txt file is a plain-text file placed at the root of a domain — accessible at `https://example.com/robots.txt` — that uses the Robots Exclusion Protocol (REP, formalized as RFC 9309 in 2022) to communicate crawling instructions to web bots. It declares which user agents may crawl which paths via `Allow:` and `Disallow:` directives, and typically also references the XML sitemap.

Robots.txt is a request, not an enforcement mechanism. Compliant crawlers — Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot, CCBot, and the major SEO tools — respect it. Non-compliant scrapers can and do ignore it. For genuine access control, sensitive paths must be protected at the application layer (authentication, IP allowlisting), not via robots.txt.

Pattern matching follows specific rules: directives match by URL prefix, `*` is a wildcard, `$` anchors the end of the URL, and the most specific rule wins per user agent. Crawlers also evaluate each user agent block independently — Googlebot does not inherit rules from the `User-agent: *` block if a `User-agent: Googlebot` block exists.

Common configuration errors include accidentally blocking critical paths (`/api/` when the API powers SSR content), forgetting to update after a CMS migration, blocking CSS or JavaScript assets that Google needs to render the page (which suppresses Core Web Vitals signals), and using robots.txt to suppress duplicate content instead of canonical tags or noindex meta tags.

Validation tools include Google Search Console's robots.txt report, the open-source `robots.txt Parser` from technicalseo.com, and Screaming Frog's crawl simulator. After every robots.txt change, audit logs should be checked for unexpected drops in crawl rate on critical URLs.