Free robots.txt Generator
Pick a starting point, decide how AI crawlers should be treated, add your own rules and get a ready-to-upload robots.txt file.
Free, with no sign-up. Fair-use limits apply. Text you paste is processed and not stored. To enforce the limits we keep any domain or address you enter and a scrambled form of your IP address for 30 days, as described in our privacy policy.
Why generate it rather than write it
A robots.txt file is plain text, but mistakes are costly. Presets give you a sensible base for common site types, and the generator writes the file in the correct order with a sitemap line and a header comment.
How to use it
Choose a starting point. Allow everything suits most sites. Block everything is for staging sites that must stay out of search. WordPress and online shop presets block the admin, cart, checkout and filter URLs that waste crawl effort. Then choose how to treat AI crawlers, add any extra paths to block or allow, list your sitemap addresses and press Generate.
AI crawlers
Some crawlers collect content to train models, others fetch pages to answer a user’s question live. You can leave them alone, block training crawlers only, block all of them, or allow them explicitly. Blocking is a request, and well-behaved crawlers honour it, but it is not a security measure.
Before you upload
Put the file at the root of your domain, at /robots.txt. Test it with the robots.txt tester and make sure your important pages are still allowed.
Frequently asked questions
Where do I put the file?
At the top level of the host it applies to, for example https://example.com/robots.txt. Each subdomain needs its own file.
Will blocking AI crawlers hurt my search rankings?
Blocking training crawlers does not affect Google Search. Blocking Googlebot itself would.
Is the output safe to use as is?
Review it first. Presets are a starting point and every site is different.
More free tools
See all free tools, or read about the Domain Tools in the dashboard.