Free tool
robots.txt generator
Create a valid robots.txt file in seconds: pick a preset or customize the rules. Copy or download it and place it in your site root.
Generated robots.txt
# robots.txt
Robots.txt generator: build your site's file without writing it by hand
Robots.txt is a text file that tells search engine crawlers which paths they may visit and which they may not. It belongs in the root of your domain: it must respond at yoursite.com/robots.txt, not in a subfolder. Put it anywhere else and Google won't find it, so it gets ignored.
There are only a few directives. User-agent names the crawler you're addressing (Googlebot, Bingbot, or the asterisk * for all of them). Disallow blocks a path, Allow reopens one inside a blocked folder, and Sitemap points to your XML sitemap URL. Example: "User-agent: *", then "Disallow: /wp-admin/" and "Sitemap: https://yoursite.com/sitemap.xml".
The costliest mistake is "Disallow: /" with user-agent *: it hides your entire site from search engines and your traffic disappears. Second mistake: blocking your CSS and JavaScript folders, so Google can't render pages and rates them worse. Third: believing robots.txt removes a page from the index. It doesn't. To pull a page out of Google you need the noindex tag; robots.txt only stops crawling, not appearance in results.
AI crawlers matter now too. GPTBot (OpenAI), ClaudeBot (Anthropic), and PerplexityBot read sites to train or to answer users. You can block them with a dedicated rule, for example "User-agent: GPTBot" + "Disallow: /", or let them through if you want visibility in AI answers.
This robots.txt generator lets you pick the rules from an interface: choose what to block, add your sitemap, decide how to treat AI bots, and download the finished file to upload to your root. No syntax to memorize and no typos. The robots.txt generator produces a valid file in a few clicks.
FAQ
Frequently asked questions
Where do I upload the robots.txt file?
In the root of your domain, the site's top folder, so it responds at yoursite.com/robots.txt. It won't work in a subfolder (yoursite.com/pages/robots.txt is ignored). Each domain and subdomain has its own separate file.
Does robots.txt actually remove a page from Google?
No. Disallow stops crawling, but a blocked page can still show up in results if other sites link to it. To drop it from the index you need the meta noindex tag on the page, which Google only reads if the page is NOT blocked in robots.txt.
Should I block AI crawlers like GPTBot and ClaudeBot?
It depends on your goals. Blocking them (User-agent: GPTBot, Disallow: /) keeps your content out of training; letting them through raises your chances of being cited in ChatGPT, Claude, or Perplexity answers. The tool lets you decide bot by bot.
What happens if my site has no robots.txt file?
Crawlers treat everything as crawlable and visit every reachable page: the site isn't penalized. The file matters when you want to exclude specific areas (admin, cart, duplicate pages) or point to your sitemap. Without special rules you can skip it, but a clean file is the safer choice.
How do I check that my robots.txt works?
Open it in your browser at yoursite.com/robots.txt to confirm it's online and readable. Then use the robots.txt Tester in Google Search Console: paste a URL and it tells you whether Googlebot is blocked or allowed, so you don't discover too late that you hid important pages.
The full audit checks indexing, sitemap, canonical and speed across the whole site, with priorities. Free, 7 days.
Full picture