robots.txt is a plain-text file served at the root of a domain (/robots.txt) that tells web crawlers which paths on a site they may or may not access. The file uses simple User-agent and Allow/Disallow directives following the Robots Exclusion Protocol, originally proposed in 1994 and now an RFC (RFC 9309).
A robots.txt rule has two parts: a User-agent line that names the bot (or wildcard *), followed by one or more Allow and Disallow lines specifying paths. The file is consulted by crawlers BEFORE they fetch the page, so a Disallow keeps the page from being requested at all.
For AEO in 2026, robots.txt is the single highest-leverage technical control. The major AI engines — OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User), Anthropic (ClaudeBot, Claude-Web, Claude-SearchBot, Claude-User), Perplexity (PerplexityBot, Perplexity-User), Google (Google-Extended for AI Overviews) — all respect robots.txt directives. A wildcard block on any of them removes you from that engine's citation candidacy.
The AEO-correct posture is an explicit "User-agent: <BotName>" + "Allow: /" stanza for every AI bot you want to be cited by, rather than relying on an implicit wildcard rule. Many sites accidentally block AI bots at the CDN or WAF layer instead of robots.txt — check Cloudflare Bot Fight Mode and similar tools alongside the file itself.