What Robots.txt Does and How to Set It Up Correctly

A robots.txt file is a plain text file placed in a website’s root directory that tells web crawlers which URLs they can and can’t request. It follows the Robots Exclusion Protocol, originally created by Martijn Koster in 1994 and formalized nearly three decades later in September 2022 when the IETF published RFC 9309. That formalization codified what had been an informal convention into an official internet standard, adding clear definitions for error handling, caching behavior, and file size limits. Every major search engine honors it, and getting it wrong can silently block revenue-generating pages from ever appearing in search results.

At Gorilla Marketing, we regularly audit robots.txt files for US businesses and find the same preventable mistakes causing real indexing problems. This guide covers the syntax, the logic, the common pitfalls, and the emerging question of AI bot control, so you can treat your robots.txt as the precision instrument it should be, not an afterthought.

What Does Robots.txt Actually Do?

Robots.txt controls crawling, not indexing. That distinction matters more than most teams realize.

When a crawler like Googlebot arrives at your domain, the first thing it requests is https://yourdomain.com/robots.txt. It reads the file, caches the directives, and then follows them when deciding which URLs to request. If a URL is disallowed, the crawler won’t fetch it. But “won’t fetch” doesn’t mean “won’t index.” If other pages link to a disallowed URL, Google can still index that URL based on anchor text and link context alone. It just won’t have the page’s content to work with.

This is the single most misunderstood aspect of the file. Blocking a URL in robots.txt doesn’t remove it from search results. It prevents Google from seeing the content on that page. If your goal is to remove a page from the index entirely, you need a noindex directive (more on that comparison below).

What robots.txt does well:

Protects crawl budget. Every site gets a finite amount of crawler attention. Blocking low-value URLs (admin panels, internal search results, staging environments) keeps crawlers focused on the pages that generate traffic and revenue.

Prevents duplicate content crawling. Faceted navigation, URL parameters, and print-friendly page versions can create thousands of near-identical URLs. Robots.txt stops crawlers from wasting time on them.

Manages server load. High-volume crawling can strain servers. Controlling access through robots.txt reduces unnecessary requests.

What it doesn’t do:

It’s not a security measure. Any human can type your robots.txt URL into a browser and read it. Listing sensitive directories in the file actually advertises their existence.

It doesn’t control indexing. A disallowed URL can still appear in search results if external links point to it.

It doesn’t pass or block link equity. Disallowing a URL doesn’t prevent PageRank from flowing to it through internal links. If you’re trying to sculpt link equity, robots.txt is the wrong tool.

Where Does the File Live and How Does Google Cache It?

The file must sit at the root of the protocol and domain combination. For https://www.example.com, the correct location is https://www.example.com/robots.txt. A robots.txt file placed in a subdirectory (like /insights/robots.txt) is ignored. Each subdomain needs its own file, so blog.example.com requires a separate robots.txt from www.example.com.

Google caches your robots.txt for approximately 24 hours. In practice, the cache duration can shift based on the max-age value in your Cache-Control HTTP headers, or extend longer if the server returns 5xx errors when Google tries to refresh the file. This caching behavior means changes aren’t instant. If you update the file and need Google to pick up the change quickly, submit it through the robots.txt report in Google Search Console to request a recrawl.

One practical implication: don’t treat robots.txt like a dynamic, real-time traffic management tool. If your server is under load, respond with a 503 or 429 HTTP status code instead. Google understands those signals immediately. Rapidly toggling robots.txt rules throughout the day won’t accomplish what you think it will, because Google may not see the changes for hours.

What About File Size?

RFC 9309 requires crawlers to parse at least 500 kibibytes (512,000 bytes) of a robots.txt file. Google follows this limit. If your file exceeds 500 KB, rules beyond that threshold may be silently ignored, and a truncated line could produce unintended behavior. For most sites, this is a non-issue. But for large enterprises with thousands of disallow rules, it’s worth checking. If you’re approaching the limit, consolidate rules using wildcard patterns instead of listing individual URLs.

How Is Robots.txt Syntax Structured?

The file is built from groups. Each group starts with a User-agent line, followed by one or more Disallow or Allow rules. Here’s a minimal example:

User-agent: Googlebot

Disallow: /admin/

Allow: /admin/public/

User-agent: *

Disallow: /staging/

Sitemap: https://www.example.com/sitemap.xml

Key Directives

User-agent identifies which crawler the rules apply to. The wildcard * matches all crawlers not called out by a more specific group. If a crawler finds a group that matches its own user-agent string, it follows that group’s rules and ignores the wildcard group entirely.

Disallow tells the matched crawler not to request URLs that start with the specified path. Disallow: /admin/ blocks /admin/, /admin/settings, /admin/users/list, and everything else under that path.

Allow overrides a broader Disallow within the same group. In the example above, /admin/public/ remains accessible to Googlebot even though /admin/ is otherwise blocked.

Sitemap declares the location of your XML sitemap. This directive isn’t tied to any specific user-agent group and applies globally. Place it at the bottom of the file.

Wildcards and Pattern Matching

Two special characters give you more control:

*** (asterisk):** Matches any sequence of characters. Disallow: /search?q=* blocks all URLs starting with /search?q=.

$ (dollar sign): Anchors the match to the end of the URL. Disallow: /*.pdf$ blocks all URLs ending in .pdf but won’t block /pdf-guide/ or /about-pdf-formats.

A few practical patterns:

Comments

Lines starting with # are comments. Use them. A robots.txt file maintained by multiple people over several years becomes unreadable without annotations explaining why each rule exists.

# Block internal search results to prevent crawl waste

Disallow: /search/

# Allow product filter pages for main categories only

Allow: /products/*/filter/

How Does Robots.txt Differ from Meta Robots and X-Robots-Tag?

These three mechanisms address different layers of crawler control. Confusing them leads to rules that conflict or cancel each other out.

The critical interaction: if you block a page in robots.txt, the crawler never fetches it, which means it never sees a noindex meta tag on that page. This is one of the most common mistakes in technical SEO. Teams add noindex to a page and then also disallow it in robots.txt, thinking they’re being thorough. In reality, the robots.txt block prevents the noindex from ever being read, and the page can still appear in search results with a URL-only listing.

Rule of thumb: Use robots.txt to manage crawl budget and prevent unnecessary fetching. Use noindex (via meta tag or X-Robots-Tag) when you need a page excluded from search results entirely. Don’t combine both on the same URL.

What Should You Block (and What Should You Leave Alone)?

Safe to Block

Internal search results pages (/search/, /?s=). These generate infinite URL combinations with thin content.

Admin and login areas (/wp-admin/, /admin/, /account/). No SEO value, and crawling them wastes budget.

Staging or development environments. If they’re accidentally accessible, robots.txt provides a first line of defense (but password protection is the real fix).

Faceted navigation parameters. Filter combinations on e-commerce sites can produce millions of crawlable URLs. Block the parameter patterns that don’t represent distinct, valuable content.

Print and PDF versions of existing pages. These are duplicate content magnets.

API endpoints exposed to the public web.

Tag and archive pages that duplicate content already available on category or main listing pages. These are common on WordPress sites and can generate hundreds of low-value URLs.

Never Block

CSS and JavaScript files. Google needs to render your pages to understand their content. Blocking CSS or JS prevents rendering, which degrades how Google evaluates the page. This was a common practice years ago and it’s still one of the most frequent mistakes we see in audits.

Pages you want indexed. Obvious, but it happens. A stray Disallow: / on a production site can deindex an entire domain.

Image directories (unless you genuinely don’t want images in Google Image Search). Blocking /images/ removes a traffic channel.

Your sitemap file. Crawlers need to fetch it. Don’t block the path.

How Should You Handle AI Crawlers in 2026?

The robots.txt file has taken on a new role in the past two years: controlling which AI companies can use your content for model training.

Most major AI providers now operate multiple user-agent strings, each serving a different purpose. Understanding the distinction is important, because you may want to block training crawlers while still allowing your content to appear in AI-powered search answers.

Key AI User-Agents (2026)

A typical configuration that blocks training while allowing AI search visibility:

# Block AI training crawlers

User-agent: GPTBot

Disallow: /

User-agent: ClaudeBot

Disallow: /

User-agent: Google-Extended

Disallow: /

User-agent: CCBot

Disallow: /

# Allow AI search retrieval

User-agent: ChatGPT-User

Allow: /

User-agent: Claude-SearchBot

Allow: /

User-agent: OAI-SearchBot

Allow: /

This approach protects your content from being absorbed into training datasets while keeping it visible when users ask AI tools questions that your pages can answer. It’s a strategic distinction, not an all-or-nothing decision.

One caveat: not all AI crawlers honor robots.txt. The major providers (OpenAI, Anthropic, Google) have committed to respecting it, but smaller or less scrupulous scrapers may ignore the file entirely. Robots.txt is a request, not an enforcement mechanism. For sensitive content, server-side access controls remain the only reliable barrier.

Should You Block AI Training Crawlers?

The answer depends on your business model. Publishers monetizing content through ads or subscriptions have a strong incentive to block training crawlers, since the content’s value diminishes if it gets absorbed into a model that can reproduce it without attribution or traffic. E-commerce sites and service businesses, on the other hand, often benefit from appearing in AI-generated answers, because those answers can drive qualified traffic.

Before making the decision, check your server logs. You may be surprised by the volume of AI crawler traffic hitting your site. Some site operators have reported that AI training bots account for a meaningful share of total bot traffic, consuming server resources without returning any direct benefit. If that’s the case, blocking training-specific user-agents reduces load and bandwidth costs while preserving your visibility in AI-powered search results.

Whatever you decide, document your reasoning in the robots.txt file itself using comments. AI crawler policies are still evolving, and six months from now you’ll want a clear record of why specific bots were blocked or allowed.

What Is Crawl-Delay, and Should You Use It?

The Crawl-delay directive tells a crawler to wait a specified number of seconds between requests:

User-agent: Bingbot

Crawl-delay: 10

Google does not support Crawl-delay. Googlebot ignores it entirely. If you need to manage Google’s crawl rate, use the Crawl Rate Settings in Google Search Console.

Bing, Yandex, and some other crawlers do respect it. If your server struggles under heavy bot traffic from non-Google crawlers, Crawl-delay can help. But for most sites with adequate hosting, it’s unnecessary and can slow down the discovery of new content.

It’s also worth noting that Crawl-delay applies per user-agent group, not globally. You can set a 10-second delay for Bingbot while leaving other crawlers unrestricted. If you do use it, monitor your server logs afterward. An overly aggressive delay (30+ seconds) can mean Bing takes weeks to discover new pages on a large site, which defeats the purpose of having content indexed quickly.

What Are the Most Common Robots.txt Mistakes?

These are the errors that show up repeatedly in technical SEO audits, often running undetected for months:

1. Blocking CSS/JS files. Legacy rules from a decade ago that prevent Googlebot from rendering pages. The result: Google can’t assess layout, above-the-fold content, or user experience signals.

2. Using robots.txt to “hide” pages from Google. The page still gets indexed if anything links to it. Use noindex instead.

3. Conflicting noindex and disallow. As covered above, blocking a URL in robots.txt prevents Google from seeing the noindex tag. Pick one method, not both.

4. Forgetting trailing slashes. Disallow: /blog blocks /blog, /insights/, /insights/post-1, and also /blogger and /blog-archive. Disallow: /insights/ only blocks paths under the /insights/ directory. That single slash changes the scope significantly.

5. No robots.txt file at all. If the file is missing, crawlers assume everything is accessible. That’s fine for small sites, but for large sites with parameter-heavy URLs, it means every possible URL combination gets crawled.

6. Blocking the sitemap path. If your sitemap lives at /sitemap.xml and you have Disallow: /*.xml$, you’ve blocked the sitemap.

7. Testing in staging, deploying to production. A restrictive robots.txt designed for a staging environment gets pushed to the live site during deployment. This is one of the fastest ways to deindex an entire site. During a site migration, checking the robots.txt file should be on every launch checklist.

8. Not accounting for subdomain scope. Rules in www.example.com/robots.txt don’t apply to shop.example.com. Each subdomain needs its own file.

9. Overly broad wildcard rules. A pattern like Disallow: /*? blocks all URLs with query strings, which sounds reasonable until you realize some CMS platforms use query parameters for essential page routing. Always test wildcard rules against actual URLs on your site before deploying them.

10. Ignoring the file after launch. Sites grow. New CMS features add URL paths that didn’t exist when the robots.txt was written. A rule that made sense two years ago might now be blocking a section that generates revenue. Periodic review isn’t optional for large sites.

How Do You Test and Validate Your Robots.txt?

Google Search Console provides a robots.txt testing tool that lets you enter a URL and check whether it’s blocked or allowed under your current rules. Use it after every change.

Beyond Google’s tool, here’s a manual validation checklist:

[ ] File returns a 200 status code (not 404 or 5xx)

[ ] File is served as text/plain content type

[ ] File is under 500 KB

[ ] No accidental Disallow: / under User-agent: *

[ ] CSS and JS directories are not blocked

[ ] Sitemap URL is declared and not blocked

[ ] Each user-agent group has at least one rule

[ ] Wildcard patterns are tested against real URLs

[ ] AI crawler rules match your content licensing strategy

[ ] No trailing whitespace or invisible characters causing parse errors

For enterprise sites, consider automating robots.txt validation in your CI/CD pipeline. A pre-deployment check that compares the staging robots.txt against the production version can catch the “accidentally pushed staging rules to production” mistake before it causes damage.

What Happens When Robots.txt Returns an Error?

The behavior depends on the HTTP status code:

200: Normal operation. Crawlers read and follow the rules.

403 (Forbidden): Google treats the entire site like it’s disallowed. No pages get crawled.

404 (Not Found): Google assumes there are no restrictions and crawls everything. This is fine for small sites, but risky for sites that relied on the file to manage crawl budget.

5xx (Server Error): Google uses its cached version of the file. If there’s no cached version and the errors persist for an extended period, Google may treat it like a 404 (open crawling) or stop crawling entirely, depending on how long the errors last.

This means your robots.txt needs the same uptime monitoring you’d give any other critical page. A server misconfiguration that accidentally returns 403 for robots.txt can halt all crawling within hours.

How Does Robots.txt Fit into Broader Crawl Management?

Robots.txt doesn’t operate in isolation. It’s one part of a crawl management strategy that also includes:

XML sitemaps, which tell crawlers what you want them to find. Where robots.txt says “don’t go here,” sitemaps say “prioritize these.” The two should work in alignment; never list a URL in your sitemap that’s blocked in robots.txt.

HTTP status codes like 301 redirects, 410 (gone), and 503 (temporary unavailability), which give crawlers real-time signals that robots.txt can’t match.

Canonical tags, which resolve duplicate content at the indexing level, handling cases where robots.txt blocking would be too blunt.

Internal linking architecture, which signals priority through link distribution rather than access control.

When crawl issues surface (pages dropping from the index, slow discovery of new SEO content, budget being consumed by low-value URLs), robots.txt is often the first file to check but rarely the only file that needs attention. Crawl errors, server response times, and redirect chains all compound the problem.

For JavaScript-heavy sites, the relationship between robots.txt and rendering deserves special attention. Google renders JavaScript to understand page content, but it needs access to the JS files and any API endpoints those scripts call. If robots.txt blocks a critical script or endpoint, the page might get crawled but render as blank or incomplete, leading to poor rankings despite being technically “indexed.”

The same principle applies to sites undergoing large structural changes. During any significant URL restructuring, the robots.txt file should be part of the migration plan, updated to reflect the new URL architecture and reviewed for rules that reference paths that no longer exist.

A Practical Robots.txt Audit Checklist

If you haven’t reviewed your robots.txt in the past six months, run through this:

Pull the file. Visit yourdomain.com/robots.txt directly. Confirm it loads, returns a 200 status, and displays as plain text.

Check the wildcard group. Look at the User-agent: * section. Are the rules intentional, or have they accumulated over time without cleanup?

Validate against your sitemap. Cross-reference every disallow rule against your sitemap URLs. Flag any overlap.

Test high-value pages. Paste your top 20 revenue-generating URLs into Google’s robots.txt tester. Confirm none are accidentally blocked.

Review AI bot rules. Decide your position on training vs. retrieval crawlers and implement the appropriate user-agent blocks.

Check for deprecated rules. Old rules blocking crawlers that no longer exist, or paths that were restructured years ago, add clutter and increase the risk of unintended matches.

Verify rendering access. Confirm CSS and JS files are accessible to Googlebot. Use Google’s URL Inspection tool to verify pages render correctly.

Document your decisions. Add comments explaining why each rule exists. Future you (or the next person managing the site) will appreciate it.

Keeping Your Robots.txt Aligned with Your Strategy

A robots.txt file isn’t something you set once and forget. It should evolve with your site’s architecture, your content strategy, and the broader search environment. New sections get launched. Old ones get deprecated. AI crawlers introduce user-agents that didn’t exist six months ago.

Build a review cadence: quarterly at minimum, and always as part of any site migration or major structural change. The file is small, the syntax is simple, and the consequences of getting it wrong are disproportionately large. A single misplaced rule can undo months of SEO work in hours, and the fix is always just a few lines of plain text.