Robots.txt Generator
Your robots.txt file tells search engine crawlers where they can and can't go. Choose your rules below, add any paths you want to block, point crawlers to your sitemap, and copy a clean, standards-compliant file.
User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /checkout
Sitemap: https://example.com/sitemap.xmlSave this as robots.txt in the root of your domain.
How to use the robots.txt tool
- 1Choose whether to allow or block crawlers by default.
- 2Add any specific paths you want to disallow (like /admin or /cart).
- 3Add your sitemap URL so search engines discover your pages faster.
- 4Copy the file and save it as robots.txt in the root of your domain.
Frequently asked questions
Where does robots.txt go?
It must live at the root of your domain - example.com/robots.txt. Crawlers only look there; a robots.txt in a subfolder is ignored.
Does disallow in robots.txt remove a page from Google?
No. Disallow stops crawling, but a blocked URL can still appear in results if other pages link to it. To keep a page out of the index, allow crawling and use a noindex meta tag instead.
Should I add my sitemap to robots.txt?
Yes. A Sitemap directive helps search engines find and crawl your important pages faster, especially on newer sites without many backlinks. You can list more than one sitemap.
Can robots.txt block a specific bot?
Yes. Add a User-agent group naming the bot (e.g. GPTBot) followed by Disallow: /. This tool has toggles for the common AI crawlers, and you can add any others manually.
What happens if my site has no robots.txt?
Crawlers assume everything is allowed. That's fine for many small sites, but a robots.txt lets you block private areas, point to your sitemap, and manage crawler access deliberately.
More free SEO tools
Skip the manual work - let Spook rank you
These tools help with the pieces. Spook does the whole job: it finds winnable queries, writes the articles, and publishes them on autopilot.