About this tool
Generates a standards-compliant robots.txt file from your crawl rules. The file is built locally in your browser.
Frequently asked questions
Where does robots.txt go?
At the root of your domain, for example https://example.com/robots.txt, so crawlers can find it.
What does Disallow do?
It asks well-behaved crawlers not to fetch the listed paths; one path per line, starting with a slash.