What Is robots.txt? A Beginner’s Guide With Examples

If you have ever looked at a website’s settings, read an SEO checklist, or copied a file from a tutorial, you have probably met robots.txt — and you may have wondered what that small text file actually does. The short answer is simple: robots.txt is a note a website leaves at its front door, telling search engine crawlers which parts of the site they are welcome to visit and which parts they should skip.

This guide explains what a robots.txt file is, how it works, what it cannot do, and the five mistakes that hurt real sites, with examples you can read line by line.

The short answer: a note on the door for crawlers

A robots.txt file is a plain text file that lives at the root of a website, at an address like example.com/robots.txt. It is written for crawlers — the automated programs search engines send out to discover pages. Google’s is called Googlebot; Bing’s is Bingbot; many other services run their own.

When a crawler arrives at a site, one of the first things it does is look for this file. If it finds one, it reads the rules before it starts fetching pages. If there is no file, the crawler simply assumes everything is open to it.

The file belongs to a system called the Robots Exclusion Protocol, standardised in 2022 as RFC 9309. “Protocol” sounds formal, but the important word is “exclusion”: the file exists to exclude crawlers from places they do not need to go, not to invite them in. Anything the file does not mention is allowed by default.

How robots.txt works

A robots.txt file is organised into groups. Each group starts with one or more User-agent lines naming the crawler the rules apply to, followed by the rules themselves. A crawler reads the group that names it most specifically; if no group names it, it follows the group for all crawlers, written as User-agent: *.

Here is the smallest useful file a site can have:

User-agent: *
Disallow: /wp-admin/

Read it as a sentence: “Every crawler, do not crawl anything under /wp-admin/.” Everything else on the site stays open, because the file says nothing about it.

Three rules of reading apply to the whole file:

• Rules match from the start of the path. A rule about /blog matches any address that begins with those characters — more on this in the mistakes section, because it catches people out.
• Rules are case-sensitive. /Admin/ and /admin/ are two different paths.
• Lines starting with # are comments. Crawlers ignore them, so you can use them to leave notes for yourself.

Google’s crawlers recognise exactly four fields: user-agent, allow, disallow and sitemap. Anything else in the file is silently ignored — which is why a made-up directive like Crawl-delay does nothing for Googlebot.

The four directives, one by one

There are only four building blocks to learn. Everything in a real robots.txt file is made from them.

User-agent: who the rules are for

User-agent names the crawler a group of rules applies to. An asterisk means “every crawler that has no more specific group”:

User-agent: *

A specific name targets one crawler only:

User-agent: Googlebot

The most specific matching group wins, and groups do not combine. If Googlebot has its own group, it follows that group alone and ignores the * group entirely — a detail that surprises people who assume the general rules always apply underneath.

Disallow: what crawlers should not fetch

Disallow lists paths a crawler should not request. A trailing slash means “this directory and everything inside it”:

Disallow: /private/

An empty Disallow blocks nothing:

Disallow:

And the single most dangerous line in the whole file:

Disallow: /

That one line blocks the entire site, because every path on the site starts with /. It has a legitimate use — keeping a staging or development site out of search results while it is being built — but it must never survive the launch. More sites have gone invisible this way than by any other robots.txt mistake.

Allow: the exception inside a block

Allow re-opens a specific path inside a blocked area. It exists so you can block a whole directory while keeping one corner of it open:

User-agent: *
Disallow: /private/
Allow: /private/public/

When a path matches both an allow and a disallow rule, the longer, more specific match wins. Allow: /private/public/ beats Disallow: /private/ for anything under /private/public/, and the rest of /private/ stays blocked.

Sitemap: where the sitemap lives

Sitemap is the odd one out: it does not belong to any group, and it takes a full web address rather than a path:

Sitemap: https://example.com/sitemap.xml

It tells crawlers where the site’s sitemap is, which helps them discover pages. A sitemap is a discovery hint, not an order — listing a page does not guarantee it will be crawled — but pointing crawlers at it costs one line and removes guesswork.

A complete example file, line by line

Here is a realistic starter file for a small online shop or blog. Read it top to bottom and it tells a coherent story:

# Every crawler, please follow these rules.
User-agent: *
Disallow: /search
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Sitemap: https://example.com/sitemap.xml

Line by line:

• User-agent: * — these rules are for every crawler.
• Disallow: /search — skip internal search results pages, which are near-duplicates that change with every query.
• Disallow: /cart/, /checkout/, /account/ — skip the shopping and account areas. There is nothing here a search visitor needs to find.
• Sitemap: https://example.com/sitemap.xml — and here is the map of the pages that do matter.

Notice what the file does not do: it never mentions the blog, the product pages, or the homepage. Those stay open by default, exactly as they should be.

What robots.txt cannot do

This is the section that saves people the most trouble, because four common beliefs about robots.txt are wrong.

It cannot keep a page out of search results. Robots.txt controls crawling, not indexing. If a page is blocked but other sites link to it, Google can still index its address — and because Google never crawled the page, the listing shows the URL with no description. That is usually worse than either outcome you wanted. If a page must stay out of results, the tool for that is a noindex instruction on the page itself — and here is the trap: a page blocked in robots.txt can never be crawled to read its noindex, so blocking a page and noindexing it at the same time cancels the noindex out. Choose one mechanism, not both. The related question of which URL search engines treat as the main version of a page is a separate mechanism again: the canonical tag, explained in our guide to what a canonical URL is.

It is not security. The file is public — anyone can open example.com/robots.txt and read it. Listing a “secret” folder there does not hide it; it advertises it. Sensitive areas need real protection, such as password login, not a disallow line.

It is not privacy. For the same reason: a disallow rule is a request, not a lock. Genuinely private content should sit behind authentication, never in a publicly listed path.

It does not bind every bot. Respectable search crawlers honour the protocol. Scrapers, spam bots and malicious crawlers may simply ignore it. RFC 9309 itself says the rules are not a form of access authorisation — they are a voluntary convention, and nothing enforces them.

Five mistakes that hurt real sites

1. Launching with the staging file. A development site is often blocked with Disallow: / so it stays out of search results while it is built. When the site goes live and nobody removes that line, every compliant crawler stays away from every page, and new pages simply never appear. Put a robots.txt check on your launch list, and check it again after every migration.

2. Forgetting that rules match by prefix. Disallow: /blog does not mean “the blog directory”. It blocks anything whose path starts with /blog — including /blog-news, /blogging-tips.html and any other page that happens to share those first five characters. If you mean the directory, write Disallow: /blog/ with the trailing slash.

3. Blocking CSS and JavaScript. Old advice sometimes suggested keeping crawlers out of style and script files, but Google needs those files to render your pages as a visitor sees them, so blocking them can make pages look broken to the crawler. Do not block your theme’s or platform’s asset folders.

4. Blocking a page and noindexing it. As covered above, the block stops Google from ever reading the noindex, so the page can stay in results indefinitely. Pick the mechanism that matches the goal: robots.txt to save crawl effort on low-value URLs, noindex (on an unblocked page) to remove a page from results.

5. Relying on rules Google ignores. Crawl-delay is not supported by Google, so it will not slow Googlebot down; if a crawler is genuinely overloading your server, the fix is server capacity or caching, not a robots.txt line. And a file in the wrong place — anything other than the root, or a file that returns a server error for a long time — may be treated cautiously or missed entirely. The file must be plain text, at the root, returning a normal success response.

Do you even need a robots.txt file?

Many small sites do not need to write one from scratch. If your site runs on a platform like WordPress, the platform already serves a sensible default. Manage My Tools itself runs on WordPress, and its live file is a good example of a minimal default:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://managemytools.com/sitemap_index.xml

That is the whole file: keep crawlers out of the admin area, let them use the one admin file the site’s features need, and point them at the sitemap. For a small blog or brochure site, a file like this — plus rules for any genuinely low-value areas of your own, such as internal search or cart pages — is all you need. Add rules when you have a real reason, not because a template told you to fill the file.

If you do change your file, test the change before it goes live. Google Search Console’s robots.txt report shows exactly how Google reads your file, and its URL Inspection tool confirms whether a specific page is blocked. A two-minute check there catches the Disallow: / disaster before your visitors do.

Build your file without typing it by hand

Writing these rules by hand is exactly how the typos creep in — a missing slash, a wrong case, a rule in the wrong group. If you would rather assemble the file than type it, our free Robots.txt Generator builds one from the paths you enter: type one allow or disallow path per line, add your sitemap address if you have one, and copy or download the finished file, ready to place at the root of your site. It is a starting point, not a finished policy — check every path against your site’s real structure before you publish it, and test the result in Search Console’s robots.txt report, since the generator assembles the file from your inputs and does not itself check a live site. You will find it among our other free SEO tools, or browse the full directory on the all tools page.

Frequently asked questions

Where does the robots.txt file go?

At the root of your domain, reachable as example.com/robots.txt. Crawlers check that exact location before they begin. A file anywhere else is not a robots.txt file as far as crawlers are concerned.

Does robots.txt hide a page from Google?

Not reliably. It stops compliant crawlers from fetching the page, but the address can still be indexed if other pages link to it — shown without a description. To keep a page out of results, use a noindex instruction on the page and leave the page crawlable so Google can read it, or protect it with a password.

Do all bots follow robots.txt?

No. Major search crawlers do, but scrapers and malicious bots often ignore it. Treat it as a request to well-behaved crawlers, never as security or access control.

What happens if my site has no robots.txt file?

Nothing bad. Crawlers assume everything is allowed and crawl normally. A missing file is a perfectly fine state for a small site; you only need a file when you have something specific to exclude or a sitemap to point at.

How do I test my robots.txt file?

Use Google Search Console’s robots.txt report, which shows how Google reads your file, and URL Inspection to check individual pages. For the official rules behind all of this, Google’s own guide to creating a robots.txt file is the reference worth bookmarking.