Web & SEO
Robots.txt Generator (incl. AI Crawlers)
Create a robots.txt file — allow or block search engines, disallow private folders, block AI training crawlers like GPTBot and ClaudeBot, and add your sitemap.
How to use
- 1Choose the default rule, folders to block and whether to block AI crawlers.
- 2Save the result as robots.txt in your site's root (example.com/robots.txt).
Frequently asked questions
Does robots.txt hide pages from Google?
It stops crawling, not indexing — a blocked URL can still appear if linked elsewhere. Use a noindex meta tag to keep a page out of results.
Can I block AI training bots?
Yes — major AI crawlers (OpenAI's GPTBot, Anthropic's ClaudeBot, Common Crawl, Google-Extended and others) honor robots.txt rules. Not every scraper does.
Where does the file go?
At the root of the domain, e.g. https://example.com/robots.txt. Subdomains need their own file.
Why blocking AI crawlers needs its own separate rule block, not a shared one
Each AI bot (GPTBot, ClaudeBot, Google-Extended, CCBot and others) gets its own 'User-agent: [bot] / Disallow: /' pair rather than being added to the wildcard User-agent: * block, because robots.txt rules are matched per user-agent string — a bot looks for a block matching its own name first, and only falls back to the wildcard block if there's no specific match for it. Lumping AI bots into the general rule would either block every crawler (including Googlebot's regular search indexing) or require search engines and AI crawlers to be indistinguishable, which they aren't.
Worked example: Robots.txt Generator
Leaving the default folders to block (/admin/, /cart/, /search) with AI blocking off and a sitemap URL set produces a four-line file: a wildcard User-agent block disallowing those three paths, followed by a blank line and the Sitemap directive. Turn on 'Block AI training crawlers' with no disallowed folders instead, and the output becomes a wildcard block that allows everything ('Allow: /'), followed by a separate Disallow: / block for each of the twelve listed AI bots by name — letting regular search engines and the human web crawl freely while explicitly telling GPTBot, ClaudeBot and the rest to stay out entirely.
Why robots.txt can't actually hide a page from search results
This is the single most common misunderstanding about robots.txt, which is why the FAQ addresses it directly: a Disallow rule stops a compliant crawler from fetching a page's content, but if another page links to that URL, search engines can still index the URL itself (showing it in results with no description, since they never crawled the content) — actually keeping a page out of search results requires a noindex meta tag on the page itself, which requires crawling it, the opposite of what Disallow does.
Why AI-crawler blocking is a request, not an enforcement mechanism
The FAQ's phrasing that major AI crawlers 'honor robots.txt rules' is doing real work — robots.txt is a voluntary convention, not a technical barrier, and it only stops bots whose operators have chosen to respect it. Reputable crawlers from OpenAI, Anthropic and Google do check and follow these rules, but a disallow line here provides no actual protection against a scraper that simply ignores robots.txt altogether, which is a meaningfully different security model than it might appear.
Why crawl-delay is included even though Google ignores it
Entering a value in the optional Crawl-delay field adds a Crawl-delay line right after the disallow rules, even though Google has stated it doesn't support that directive at all — it's a holdover respected mainly by Bing, Yandex and some smaller or more aggressive crawlers that can otherwise hit a small server hard with rapid-fire requests. Leaving the field blank omits the line entirely rather than writing a zero or default value, so a site that doesn't need to throttle any crawler gets a cleaner file with no unnecessary directive sitting in it.
Related tools
- Press Release Template Generator
Draft a properly formatted press release — headline, dateline, lead paragraph, quote, boilerplate and media contact — from a few details.
- Schema Markup (JSON-LD) Generator
Generate JSON-LD structured data for FAQ pages, articles, products, local businesses and organizations — ready to paste into your page's <head>.
- Hreflang Tag Generator
Generate hreflang link tags (and the x-default) for multilingual or multi-regional sites, as HTML <link> tags or XML sitemap entries.
- Email List Cleaner & Validator
Clean an email list in seconds — remove duplicates and invalid addresses, fix typos like gmial.com, and optionally drop role (info@, noreply@) and disposable addresses. Paste any list; nothing is uploaded.
- AI Prompt Generator & Improver
Turn a rough request into a clear, structured prompt for ChatGPT, Claude or Gemini — with role, context, task, constraints and output format.
- Nginx Config Generator
Generate an nginx server block for a static site, SPA or reverse proxy — with HTTPS (Let's Encrypt paths), HTTP→HTTPS and www redirects, gzip and caching.