Free tool
Robots.txt generator
Build a correct robots.txt without memorising the syntax. Set your crawler defaults, block individual AI bots, add your sitemap and copy the finished file — it updates live as you click.
Your robots.txt
User-agent: * Disallow:
Save this as a file named robots.txt and upload it to the root of your site so it is reachable at yourwebsite.com/robots.txt.
How to use
Three steps, ten seconds
- 01
Choose your defaults
Allow everything (the right choice for almost every site) or block everything — useful for staging sites you want kept out of Google.
- 02
Fine-tune the rules
Tick any AI crawlers you want to block, paste your sitemap URL and list private paths like /admin/ — one per line.
- 03
Copy and upload
The file builds live as you click. Copy it, save it as robots.txt and upload it to your site's root folder.
FAQ
Common questions
What does robots.txt actually control?
robots.txt tells well-behaved crawlers which parts of your site they may fetch. It lives at yourwebsite.com/robots.txt. Note it controls crawling, not indexing — a blocked page can still appear in Google if other sites link to it. To keep a page out of search results entirely, use a noindex meta tag instead.
Should I block AI bots like GPTBot and ClaudeBot?
It is a trade-off. Blocking GPTBot, ClaudeBot, CCBot and PerplexityBot stops your content being used for AI training and answers, but it may also remove you from AI-powered search results that increasingly send real customers. Many local businesses leave AI bots allowed for visibility; publishers protecting original content often block them.
Where do I put the robots.txt file?
Upload it to the root of your domain so it loads at yourwebsite.com/robots.txt — subfolders do not work. On WordPress, SEO plugins like Yoast can write it for you; on most hosts you just drop the file into the public_html (web root) folder via the file manager or FTP.
More free tools
Keep the momentum going
.htaccess Redirect Generator
Turn a list of old and new URLs into 301 redirect rules for Apache or nginx in seconds.
Open tool →Meta Tag Generator
Write title and description tags, preview the Google result and copy the HTML.
Open tool →SERP Preview
See exactly how your page appears in Google on desktop and mobile before you publish.
Open tool →Free website audit
Want the full picture?
Tools fix one thing at a time. Our free audit checks your whole site — speed, SEO, mobile, security — and tells you exactly what is costing you customers.
Run the free 20-second scan →