Robots.txt Generator

Base

Extra disallow rules

One path per line.

Sitemap URL

Preferred host

Crawl delay, seconds

Result

The robots.txt fileUser-agent: * Disallow: /admin/ Disallow: /account/ Disallow: /cart/ Disallow: /search Allow: / Disallow: /tmp/ Disallow: /private/ Sitemap: https://example.com/sitemap.xml
Baseblock utility sections
Lines9
Sitemap lineSitemap: https://example.com/sitemap.xml
What mattersCheck the file in a webmaster console after publishing: one stray Disallow closes an entire section.

It forbids crawling, not indexing

Builds a robots.txt file with the disallow rules you need, a sitemap line, a preferred host and a crawl delay.

The great misunderstanding about robots.txt: it does not remove pages from an index. Disallow stops a crawler visiting a page, but if external links point at it the engine may still list it without a description — 'no information is available for this page'.

To actually remove a page you need a noindex tag or a 410, and the crawler has to be able to read them. A page blocked in robots.txt with noindex inside stays in the results, because the crawler never sees the tag.

What this calculator shows

The finished file with your chosen base and your own disallow rules
The sitemap line and the preferred host
A warning about what robots.txt does not do

What to keep in mind

The file sits strictly at the domain root and applies only to that domain and protocol.
Google ignores Crawl-delay; crawl rate there is adjusted in the webmaster console instead.

FAQs

Does robots.txt remove a page from the index?

No. It forbids crawling, but the page can stay in the results without a description. Removal needs noindex or a 410 on a page the crawler can reach.

What should always be blocked?

Internal search, cart, account pages, and sorting or filter parameters unless they are optimised as landing pages. All of them breed duplicates.

Is the Host directive needed?

Not any more: Yandex dropped it in 2018 in favour of a 301 to the preferred host. Leaving it in is harmless but pointless.

Where does the file go?

Strictly at the root: example.com/robots.txt. A file in a subdirectory is not read, and subdomains need their own.

Worked example

A typical shop configuration

Input: Block utility sections, plus /tmp/ and /private/, sitemap given

Output: Nine lines of disallows, an allow, and a sitemap

Note: The cart, account area, admin and internal search are closed — precisely the sections that generate endless duplicates. The catalogue stays fully open.

Useful links

All tools in Dev Tools

Robots.txt Generator

Result

Related tools

Redirect Rule Generator

HTTP Status Code Lookup

Meta Tag Length Checker

Useful links