Free Robots.txt Generator

What is robots.txt and why does it matter?

robots.txt is a plain text file served from the root of your domain that implements the Robots Exclusion Protocol (REP). When a crawler reaches your site, it requests https://yourdomain.com/robots.txt before any other URL and uses the rules inside to decide which paths it may fetch. The protocol is a public convention, not an enforced standard, which means well-behaved bots like Googlebot, Bingbot, and most legitimate SEO tools obey it, while malicious scrapers can ignore it entirely.

The file is a request, not a wall. It does not block humans, it does not restrict access at the HTTP level, and it does not provide any security guarantee. Anyone can fetch your robots.txt directly in a browser and read every path you have listed. Treat it as a public document. If a path is sensitive, protect it with authentication, not with a Disallow line.

robots.txt also does not remove pages from search results. A URL that is disallowed in robots.txt can still appear in Google's index if other sites link to it. Google will show the URL with no snippet, since it cannot fetch the page to read its contents. To keep a page out of the index entirely, leave it crawlable and add a noindex directive via meta tag or X-Robots-Tag header.

The most common production catastrophe with robots.txt is a single line: Disallow: /. This blocks every path on the site. It is the default in many staging environments to prevent crawlers from indexing development URLs, and it gets pushed to production by accident more often than any SEO would like to admit. Within hours, organic traffic begins to fall as Google drops pages from the index. Always diff the production robots.txt against staging before each deploy, and treat the file as a tracked artifact in version control.

How to use this generator

The generator is designed to take you from a blank file to a validated, production-ready robots.txt in under a minute.

Step 1 — Pick a CMS preset. Choose the platform that matches your stack: WordPress, Shopify, Next.js, Joomla, or start with a blank template. Each preset loads the standard set of rules for that platform, including the directories that are typically disallowed and the sitemap location commonly used by that CMS.

Step 2 — Customize the rules. Add or remove Allow and Disallow paths, group rules by User-agent (use * for all crawlers, or target specific bots like Googlebot or Bingbot), and append a Sitemap directive pointing at your sitemap URL. If you need to throttle a specific crawler, add a Crawl-delay line, but be aware that Googlebot ignores it.

Step 3 — Validate against your live site. Enter your domain and the validator fetches your existing /robots.txt, parses both files, and shows the diff. You see exactly which paths change status, which rules conflict, and whether your sitemap reference resolves. Only then do you copy the output and deploy it.

The output is plain text, formatted exactly as Google's parser expects:

User-agent: *
Disallow: /admin/
Allow: /admin/public/
Sitemap: https://example.com/sitemap.xml

Common patterns by CMS

WordPress

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-includes/
Sitemap: https://example.com/sitemap_index.xml

The wp-admin directory contains the admin panel and should not be crawled, but admin-ajax.php is used by front-end features such as theme search and comment submission, so it is explicitly allowed. wp-includes holds core PHP files that have no SEO value. The sitemap path sitemap_index.xml is the default for both Yoast SEO and Rank Math, which generate the file dynamically; if you use a different SEO plugin or a static sitemap, adjust the URL.

Shopify

User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /orders
Disallow: /checkout
Disallow: /search

Shopify ships a default robots.txt that already blocks /admin, /cart, /orders, and /checkout to keep transactional and account pages out of search results. Since 2021, Shopify lets you override the file by creating a robots.txt.liquid template in your theme. Disallowing /search is optional but common, since search result pages produce thin, near-duplicate content that can dilute your crawl budget.

Next.js / Vercel

User-agent: *
Disallow: /api/
Disallow: /_next/
Sitemap: https://example.com/sitemap.xml

In the App Router, the idiomatic approach is to generate robots.txt programmatically with app/robots.ts, which exports a MetadataRoute.Robots object. A static file in /public/robots.txt also works and takes precedence if both exist. Disallow /api/ to keep route handlers out of search and /_next/ to skip build artifacts. On Vercel, preview deployments are exposed at unique subdomains; serve a separate Disallow: / robots.txt on preview environments to prevent them from being indexed.

Joomla

User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /components/
Disallow: /modules/

Joomla ships with a default robots.txt.dist file that you copy to robots.txt on first deploy. The standard rules block the admin panel, cache directory, and core component and module folders. Newer Joomla versions auto-generate sitemaps via extensions like JSitemap or OSMap, and you should append the resulting URL with a Sitemap: directive.

Common mistakes that hurt SEO

Most robots.txt damage is self-inflicted. The patterns below cause the bulk of organic-traffic incidents we see in audits.

Blocking CSS and JS. Google has rendered pages with a real Chromium engine for over a decade, which means it needs to fetch your stylesheets and scripts to understand how your page actually looks. Blocking /css/, /js/, or a build directory like /_next/static/ causes Googlebot to render a broken layout. Mobile-friendliness, layout shift, and content visibility all degrade, and rankings can drop accordingly. Allow these directories explicitly if your CMS blocks them by default.

Leftover staging blocks. A Disallow: / line on production is the single most common cause of overnight traffic loss. It usually arrives because a developer copied a staging config without updating the environment-specific values. Use environment variables to set the robots policy, generate the file at build time, and make staging robots a separate artifact that physically cannot ship to production.

Conflicting Allow and Disallow. When two rules in the same User-agent group cover the same path, Google uses the most specific rule by path length. Older or simpler crawlers process rules top-to-bottom and take the first match. The two strategies disagree often enough to cause real-world surprises, so write rules so the outcome is the same either way: list more specific Allow lines first, then broader Disallow lines.

Blocking your sitemap. It happens when a site disallows /sitemap/ and the sitemap actually lives at /sitemap.xml. The directive Disallow: /sitemap/ does not match the file /sitemap.xml, but it is close enough that copy-paste errors create matches such as Disallow: /sitemap. Always verify the live sitemap URL is reachable after deploying robots.txt.

Misusing Crawl-delay. Googlebot ignores Crawl-delay. Bingbot honors it. Setting Crawl-delay: 30 on a large site tells Bing to fetch one URL every thirty seconds, which means Bing can crawl roughly 2,880 URLs per day. For a site with 100,000 URLs, that is over a month between full crawls, and freshness drops. Use the Bing Webmaster Tools crawl-rate setting if you actually need to throttle, and leave Crawl-delay out otherwise.

Security: don't rely on robots.txt for secrets

robots.txt is publicly readable. Anyone can open https://yourdomain.com/robots.txt in a browser and see every path you have listed. Attackers know this and routinely fetch robots.txt as a reconnaissance step, because the file often points directly at admin panels, internal tools, backup directories, and pre-launch sections.

Listing Disallow: /admin/ does not hide your admin panel; it advertises that an admin panel exists at /admin/. The same applies to Disallow: /staging/, Disallow: /old-site/, or any path you would prefer attackers not discover. If a path needs to stay private, put it behind HTTP authentication, IP allow-listing, or a real authorization layer. The web server should return 401 or 403 to unauthenticated requests, regardless of what robots.txt says.

When you want to keep a page off Google but the URL itself is not sensitive, use the noindex directive. Add <meta name="robots" content="noindex"> in the page <head>, or send an X-Robots-Tag: noindex HTTP response header. Crawlers must be able to fetch the page to see the directive, so leave the path crawlable in robots.txt. The combination — crawlable plus noindex — keeps the URL out of the index without revealing its location to anyone reading robots.txt.