Robots.txt Generator

Written byOleh Soloninupdated 5 October 2026

Base

Extra disallow rules

One path per line.

Sitemap URL

Preferred host

Crawl delay, seconds

Google does not support this line and Yandex has ignored it since 22 February 2018; Bing, for one, still reads it.

Result

The robots.txt file
User-agent: * Disallow: /admin/ Disallow: /account/ Disallow: /cart/ Disallow: /search Allow: / Disallow: /tmp/ Disallow: /private/ Sitemap: https://example.com/sitemap.xml
Baseblock utility sections
Lines9
Sitemap lineSitemap: https://example.com/sitemap.xml
What matters
Check the file in a webmaster console after publishing: one stray Disallow closes an entire section.

It forbids crawling, not indexing

Builds a robots.txt file with the disallow rules you need, a sitemap line, a preferred host and a crawl delay.

The great misunderstanding about robots.txt: it does not remove pages from an index. Disallow stops a crawler visiting a page, but if external links point at it the engine may still list it without a description — 'no information is available for this page'.

To actually remove a page you need a noindex tag or a 410, and the crawler has to be able to read them. A page blocked in robots.txt with noindex inside stays in the results, because the crawler never sees the tag.

How to use

Pick a starting point: open the site, block the housekeeping sections, or block everything.
Add your own rules — one path per line, each starting with a slash.
Give the sitemap as a full address; a relative path does not work on that line.
Upload the file to the root of the domain as robots.txt — nobody looks for it in a subfolder.

What each starting point produces

The three presets differ in line count and in meaning. "Open everything" writes an empty Disallow, which is a statement rather than an absence. "Block housekeeping" adds four typical paths and an allow line. "Block everything" writes a single Disallow: / and a warning to go with it.

That warning matters more than the file. A page blocked by Disallow does not leave the index: the crawler stops fetching it, but external links can still put it in the results — with no description, because reading it was forbidden.

The line counts come from the generator on this page with the two extra rules it ships with. The housekeeping preset does not replace your lines; it adds to them, and the file is assembled from both halves.
Starting point	Lines	What lands in the file
Open the whole site	5	User-agent, an empty Disallow, your rules, Sitemap
Block housekeeping sections	9	the same plus /admin/, /account/, /cart/, /search and Allow: /
Block the whole site	5	Disallow: / and a warning that this is not removal from the index

Which lines exist, and who reads them

The robots.txt language is short: five kinds of line and no more. Everything beyond that circulating in guides either stayed a proposal or is ignored by Google.

The last row is the trap. Noindex in robots.txt was never a supported rule, and Google stopped processing it on 1 September 2019. Removing a page from the index needs a robots meta tag or an X-Robots-Tag header — and for either of those, the page has to stay crawlable.

These are exactly the lines the generator writes, minus the last one. Noindex is not written at all, because it never belonged there in the first place.
Line	What it does	Google	Bing
User-agent: *	which crawlers the following lines apply to	honours it	honours it
Disallow: /path	the crawler does not fetch that path	honours it	honours it
Allow: /path	an exception inside a blocked area	honours it	honours it
Sitemap: https://…	where the sitemap lives	honours it	honours it
Crawl-delay: 5	a pause between fetches	not supported	ignored since 2018
Noindex: /path	never was part of the standard	ignored	ignored

What this calculator shows

The finished file with your chosen base and your own disallow rules
The sitemap line and the preferred host
A warning about what robots.txt does not do

What to keep in mind

The file sits strictly at the domain root and applies only to that domain and protocol.
Google does not support Crawl-delay and Yandex has ignored it since 2018; Bing, for one, reads it.

FAQs

Does robots.txt remove a page from the index?

No. It forbids fetching, not showing. A blocked page with external links pointing at it can appear in results with no description, precisely because the crawler was not allowed to read it. Removal needs noindex or a 410, and both require the page to stay crawlable.

What should always be blocked?

Housekeeping sections, the cart, the account area and on-site search results: they are effectively endless, useless in the results and expensive in crawl budget. Everything else gets blocked for a stated reason, not as a precaution.

What does an empty Disallow line mean?

That nothing is blocked. In effect it matches having no file with a 404 — neither blocks crawling — but an empty Disallow says "crawl everything" explicitly, and that is what a webmaster console will report.

Is the Host directive needed?

Not any more. Google never supported it, and Yandex retired it in 2018 in favour of a 301 and a canonical link. The generator only writes the line if you fill the field in yourself.

Does Crawl-delay work?

In Bing yes. Google does not support the line, and it retired the crawl-rate setting in Search Console on 8 January 2024; Yandex has ignored Crawl-delay since 22 February 2018 and sets crawl speed in its Webmaster tools instead. If the server genuinely cannot keep up, answering 429 or 503 with a Retry-After header is more reliable than this line.

Where does the file go?

In the root of the domain, at https://example.com/robots.txt. Nobody looks in a subfolder. A subdomain has its own file, and http and https are formally separate files even when the same server answers both.

Worked example

A typical shop configuration

Input: Block utility sections, plus /tmp/ and /private/, sitemap given

Output: Nine lines of disallows, an allow, and a sitemap

Note: The cart, account area, admin and internal search are closed — precisely the sections that generate endless duplicates. The catalogue stays fully open.

Useful links

All tools in Dev Tools

Robots.txt Generator

Result

Related tools

Redirect Rule Generator

HTTP Status Code Lookup

Meta Tag Length Checker

Useful links