Robots.txt: The Complete Guide with Real-World Examples

Stop guessing with your robots.txt file. Our complete guide breaks down the syntax, directives, and real-world examples you need to control web crawlers effectively.

What is Robots.txt and Why Should You Care?

Let’s get one thing straight: your `robots.txt` file is not a security measure. It’s a public text file that provides polite suggestions to web crawlers about which parts of your site they should or shouldn’t access. This is the first, and most important, lesson in our robots.txt guide.

Think of it as the bouncer at your website’s front door. It can tell Googlebot, Bingbot, and other well-behaved bots where they aren’t welcome. Malicious bots, on the other hand, will ignore it completely and try to sneak in through the kitchen.

The file lives in the root directory of your domain (e.g., `yourdomain.com/robots.txt`) and follows the Robots Exclusion Protocol (REP). Its primary function is to manage crawler traffic, not to prevent pages from appearing in search results. That’s a critical distinction we’ll beat into the ground shortly.

When you run a crawl with ScreamingCAT, it respects these rules by default. This ensures your audit mimics how search engines access your site, giving you a more accurate picture of what they see.

The Anatomy of a Robots.txt File: Syntax and Directives

A `robots.txt` file is deceptively simple. It consists of groups of directives, with each group applying to a specific `User-agent`. You can have multiple groups for different bots, or one that applies to all.

The syntax is rigid. One directive per line, no extra characters, and watch your spelling. A single typo can render a rule useless or, worse, block your entire site.

Here are the core components you’ll work with:

User-agent: *
# This is a comment. It applies to all user agents.
Disallow: /private/
Disallow: /tmp/
Disallow: /search?q=*

User-agent: Googlebot
# Only applies to Google's main crawler.
Allow: /private/google-only-content.html

Sitemap: https://www.yourdomain.com/sitemap.xml

• User-agent: Specifies the crawler the following rules apply to. `User-agent: *` is a wildcard that targets all bots.
• Disallow: The path following this directive should not be crawled. A blank `Disallow:` means you’re allowing everything.
• Allow: This directive, primarily supported by Google and Bing, overrides a `Disallow` rule within the same user-agent block. It’s useful for allowing access to a specific file or subfolder within a disallowed directory.
• Sitemap: Provides the absolute URL of your XML sitemap. You can include multiple sitemap directives.
• Wildcards: The `*` character matches any sequence of characters, and the `$` character signifies the end of a URL path.

A Practical Robots.txt Guide: Common Use Cases and Mistakes

Theory is great, but applying it is what matters. This section of our robots.txt guide covers real-world scenarios and the catastrophic mistakes we see all too often.

A common use case is blocking faceted navigation parameters to prevent crawlers from finding millions of duplicate, low-value URLs. You can block any URL containing a `?` by using `Disallow: /*?*`. This simple line can save you a world of crawl budget headaches.

Another classic is blocking staging or development environments. The last thing you want is your half-finished `dev.yourdomain.com` getting indexed. A simple `User-agent: *` followed by `Disallow: /` on that subdomain’s `robots.txt` file handles it.

However, the most common mistake is confusing crawling with indexing. Blocking a page in `robots.txt` does not guarantee it will be removed from Google’s index. If the page has been linked to externally, Google can still find and index it without ever crawling the content.

A Robots.txt Guide for Crawl Budget Optimization

Your crawl budget is the number of pages search engines can and want to crawl on your site within a given timeframe. It’s a finite resource, and `robots.txt` is your primary tool for managing it.

By disallowing crawlers from accessing low-value sections, you guide them toward the pages that actually matter. Think internal search results, filtered views, user profiles, or admin login pages. These areas offer zero value to search users and waste precious crawl budget.

Running a site audit with ScreamingCAT can quickly reveal URL patterns that generate thousands of non-indexable, parameter-driven pages. Once you identify these patterns, you can add corresponding `Disallow` rules to your `robots.txt` to cut off the waste.

A well-crafted `robots.txt` ensures that when Googlebot visits, it spends its time on your core product pages and insightful blog posts, not an infinite calendar from 2003.

Advanced Directives, Testing, and Final Checks

While `User-agent`, `Disallow`, and `Allow` do most of the heavy lifting, a few other directives exist. `Crawl-delay`, for example, was once used to slow down crawlers, but Googlebot no longer respects it. Google adjusts its crawl rate automatically based on server response.

The most crucial step before deploying any changes is testing. A single misplaced slash can have devastating consequences. Use Google Search Console’s robots.txt Tester to validate your syntax and test specific URLs against your rules.

For more complex scenarios, ScreamingCAT offers a powerful feature. In the configuration settings, you can specify a custom `robots.txt` file. This lets you run a test crawl using your proposed changes to see exactly what would be blocked *before* you push it live and accidentally de-index your entire site.

Finally, always include a link to your XML sitemap. It helps search engines discover all your important URLs, complementing the exclusion rules you’ve set. It’s a simple, effective way to tie your crawling instructions together.