robots.txt generator

What robots.txt can and cannot do

robots.txt is a request, not a lock. Well-behaved crawlers read it and obey. Badly behaved ones read it to find out where the interesting directories are. It is a crawl-budget tool, not a security tool.

Disallow does not mean deindex

This is the single most misunderstood thing about the file, and it costs people real traffic.

Disallow stops a crawler fetching a URL. It does not remove that URL from search results. If other pages link to it, Google can still list it — as a bare URL with no title and no description, because it was never allowed to look at the page.

Worse, blocking a page in robots.txt means the crawler never sees a noindex tag on it. The two instructions fight, and the block wins.

The meta tag generator produces the noindex tag when you need one.

Syntax rules worth knowing

• The file must be at the domain root — /robots.txt, nowhere else. A file in a subfolder is ignored entirely.
• It applies per host and per protocol. https://example.com and https://shop.example.com need their own files.
• Paths are case-sensitive. /Admin/ does not block /admin/.
• A trailing slash matters: Disallow: /private blocks /private-notes.html too. Disallow: /private/ blocks only the folder.
• * matches any sequence, $ anchors the end. Disallow: /*.pdf$ blocks PDFs.
• A crawler obeys the single most specific matching user-agent block and ignores all the others — so a rule you put under User-agent: * does not apply to a bot that has its own block.
• An empty Disallow: means allow everything. Disallow: / means block everything. One character between those two.

Never block your CSS or JavaScript

Google renders pages the way a browser does. If your stylesheet or scripts are blocked, it sees an unstyled document, cannot judge mobile-friendliness, and may not see content your JavaScript inserts.

Blocking /assets/ or /wp-includes/ was standard advice in 2012 and is actively harmful now. Search Console's URL Inspection tool shows you the rendered page — if it looks broken there, something is blocked.

The AI crawler question

There is no settled answer, and anyone telling you there is has a position to sell. The trade-off:

Blocking keeps your writing out of training sets and saves some server load. For a site whose content is the product — original research, a paid archive — that can be the right call.

Allowing keeps you eligible to be cited when an assistant answers a question in your area. That referral traffic is small today and growing.

The two are also not one decision. GPTBot and Google-Extended are training crawlers; OAI-SearchBot and PerplexityBot fetch pages to answer a live query and cite them. Blocking the first group and allowing the second is a coherent position. The tick box above blocks both, so edit the output if you want the split.

Testing it

Search Console has a robots.txt report that shows the file Google last fetched and flags syntax errors. Use URL Inspection on a specific page to confirm whether it is blocked.

A mistake here is unusually expensive. A stray Disallow: / pushed to production can take a site out of search within days, and recovery is slower than the fall. Check it after every deploy that touches the file — and if you use the staging preset above, make very sure it does not ship.

Questions about this generator

Related tools

Learn the why, not just the how

Longer reading on formatting, indentation and minification.