Robots.txt Generator
Build a valid robots.txt file with a simple point-and-click rules editor — no syntax to memorize, free to download.
A robots.txt file tells search engine crawlers which parts of your site they may visit. This builder lets you add user-agent rules one by one — allow or disallow any path — attach your sitemap URL, and generate a clean, standards-compliant file. Download it and upload it to your site's root folder.
How to use the Robots.txt Generator
- Add one row per rule: pick the user-agent (or * for all bots), choose Allow or Disallow, and type the path.
- Add your sitemap URL if you have one, then click Generate Robots.txt to preview the file.
- Copy the rules or click Download to save robots.txt, then upload it to your domain's root folder.
Robots.txt Generator — FAQs
What is a robots.txt file?
It is a small text file placed at the root of your website, for example example.com/robots.txt, that gives instructions to search engine crawlers. It tells bots which pages or folders they are allowed to crawl and which they should skip, helping you control how your site is indexed without editing any code.
Will robots.txt hide my private pages from everyone?
No. Robots.txt is a request, not a lock — well-behaved search bots obey it, but it does not block hackers, scrapers or curious visitors. Never use it to protect passwords, admin panels or personal data. For genuinely private content, use proper authentication, not robots.txt.
What does the * user-agent mean?
The asterisk is a wildcard that matches every crawler, so rules under 'User-agent: *' apply to Google, Bing and all other bots at once. If you want special rules for one crawler, such as Googlebot, add a separate row with that exact user-agent name and its own allow or disallow entries.
Is my data sent anywhere when I use this tool?
No. The generator runs completely in your browser. Your paths, sitemap URL and rules never leave your device and are never stored on our servers. Everything is generated locally and the download is created right on your own machine.