Robots.txt Generator
Create a robots.txt file to control how search engines crawl your site, with correct syntax. No signup. Ideal for website owners managing crawler access.
Result
| robots.txt |
|---|
Share on Social Media:
A robots.txt generator makes the small text file that tells search engines which parts of your site they may crawl — instantly, free, with no account. This one runs entirely in your browser: pick Allow all or Disallow all, add the paths to block or allow, add your sitemap address if you have one, and it builds the file for you to copy. There is no download button, so you copy the text, save it as robots.txt, and upload it to your site’s root folder.
On this page
- How to Generate a robots.txt File Step by Step
- What robots.txt Is and What It Does
- Generating Versus Writing robots.txt by Hand
- Who Uses a robots.txt Generator
- Pro Tips for robots.txt
- Common robots.txt Mistakes to Avoid
- Related Tools You May Need Next
- Frequently Asked Questions
How to Generate a robots.txt File Step by Step
- Choose the default access. Leave it on Allow all if crawlers may visit most of your site, or switch it to Disallow all if you are blocking the whole site — for example a staging copy that must never appear in search results.
- List the paths to disallow, one per line. Type each path starting with a slash, for example
/wp-admin/or/cart/. Leave this box empty if there is nothing to block. - List any paths to allow, one per line. Use this for an exception inside a blocked folder, such as allowing
/wp-admin/admin-ajax.phpwhile blocking the rest of/wp-admin/. - Add your sitemap address (optional). Paste the full address, for example
https://example.com/sitemap.xml, so crawlers can find your sitemap from the same file. - Click Generate robots.txt. The result appears below the form at once. Click the copy button, paste the text into a plain-text editor, save it as
robots.txt, and upload it to your site’s root folder — the same folder as your homepage, so it is reachable atyourdomain.com/robots.txt.
Worked example — a small blog that wants crawlers everywhere except the admin area, the cart, and its search pages, with one exception for admin-ajax.php. Enter Allow all, disallow /wp-admin/, /cart/, /search/, allow /wp-admin/admin-ajax.php, and add the sitemap https://example.com/sitemap.xml. The tool produces exactly this, 6 lines and 144 characters:
| Line | robots.txt line |
|---|---|
| 1 | User-agent: * |
| 2 | Disallow: /wp-admin/ |
| 3 | Disallow: /cart/ |
| 4 | Disallow: /search/ |
| 5 | Allow: /wp-admin/admin-ajax.php |
| 6 | Sitemap: https://example.com/sitemap.xml |
Notice the Allow line comes after all the Disallow lines, even though it is an exception to /wp-admin/. That is how the tool always orders the file, and it is correct: when a page matches both an Allow and a Disallow rule, the longer, more specific rule wins.
A second example — blocking a whole staging site. Pick Disallow all and leave everything else empty:
| Line | robots.txt line |
|---|---|
| 1 | User-agent: * |
| 2 | Disallow: / |
And if you change nothing at all and just click Generate, you get the allow-everything file:
| Line | robots.txt line |
|---|---|
| 1 | User-agent: * |
| 2 | Disallow: |
The empty Disallow: line means “disallow nothing”, so every crawler may visit every page. It looks unfinished, but it is valid.

What robots.txt Is and What It Does
A robots.txt file is a plain-text file placed at the root of a website that tells search engine crawlers which parts of the site they may or may not crawl. Crawlers check for it at yourdomain.com/robots.txt before they crawl, and the ones that respect it follow its rules. It is a set of instructions, not a lock: it guides well-behaved crawlers, it does not physically stop anyone from opening a page.
This generator can put exactly three kinds of lines in your file:
| Line the tool makes | What it means |
|---|---|
User-agent: * | The rules that follow apply to all crawlers. The tool always writes this one line, first, and you cannot change it. |
Disallow: /path/ and Allow: /path/ | One line per path you type, Disallow lines first, then Allow lines. A path starting with / is a folder or page on your site. |
Sitemap: https://example.com/sitemap.xml | Added last, only if you fill in the sitemap box. It tells crawlers where your sitemap lives. |
What this tool does not make: there is no way to write rules for one crawler only, such as Googlebot but not Bingbot — every file it makes starts with User-agent: * for all crawlers. There is also no Crawl-delay setting, no Host line, and no per-bot groups. If you need those, add them by hand to the copied text before you upload it.
The tool also does not check what you type. A path without its leading slash, a full web address in a Disallow line, or a made-up address in the Sitemap box is copied into the file exactly as you typed it, with no warning. Always read the result before you upload it, and type every path starting with /.
Generating Versus Writing robots.txt by Hand
| Aspect | Using this generator | Writing by hand |
|---|---|---|
| Line format | Always correct: User-agent: *, then Disallow: / Allow: lines, then Sitemap: | Easy to mistype a directive or forget the colon |
| Line order | Fixed and correct every time | Easy to put the Sitemap line or an Allow line in the wrong place |
| Your paths and addresses | Copied exactly as typed — a typo goes straight into the file | Same risk, but you see each line as you write it |
| Per-crawler rules | Not possible: always User-agent: * | Possible, if you know the syntax |
| Crawl-delay and other directives | Not offered | Possible, if you know the syntax |
| Speed | Seconds, in the browser, no account | Minutes, and you need to remember the format |
| Best for | The common cases: allow all, block a few folders, block a whole site, add a sitemap | Unusual cases, such as different rules for different crawlers |
The honest summary: the generator guarantees the shape of the file, not the sense of it. It will never misspell Disallow or forget the User-agent line, but it will faithfully copy a wrong path. Check the output, then upload it.
Who Uses a robots.txt Generator
Website owners. Creating a robots.txt for a new site so search engines crawl the pages that matter and skip the rest.
People managing SEO. Keeping crawlers out of admin areas, carts, checkout pages, and internal search results so the crawl budget goes to real content.
Anyone launching or staging a site. Putting a Disallow-all file on a staging copy so it never shows up in search results before launch.
Blog and shop owners. Blocking /wp-admin/ or /cart/ while still allowing the one file inside that the site needs, such as admin-ajax.php.
People pointing crawlers to a sitemap. Adding the Sitemap: line so crawlers find new pages faster.

Pro Tips for robots.txt
Start every path with a slash. Write /admin/, not admin/. Without the slash the rule does not match the folder you meant, and the tool will not warn you.
Block folders, not single pages, where you can. Disallow: /cart/ covers every page under the cart in one line.
Leave the path boxes empty when you pick Disallow all. Extra Disallow lines are still added after Disallow: /, which is redundant — the whole site is already blocked.
Keep the sitemap line in. If your site has a sitemap, add its full address. It costs one line and helps crawlers discover new pages.
Test the file after uploading. Open yourdomain.com/robots.txt in a browser to confirm it is live, and check it in Google Search Console’s robots.txt tester to confirm your rules do what you intended.
Never use robots.txt for security. It only asks well-behaved crawlers to stay away. Anything truly private needs a password or proper access control, not a Disallow line — and remember the file itself is public, so anyone can read it and see which folders you blocked.
Common robots.txt Mistakes to Avoid
Accidentally blocking the whole site. One Disallow: / hides every page from search results. This is the most damaging robots.txt mistake there is, and it usually happens when someone copies a staging file to the live site. Check the first two lines of the output before you upload.
Typing a path without its leading slash. Disallow: admin/ does not block /admin/. The tool copies it without complaint, so proofread every line.
Putting a full web address in a Disallow line. Disallow lines take paths such as /private/, not addresses such as https://example.com/private/. Full addresses belong only in the Sitemap box.
Blocking a file the site needs. Blocking /wp-admin/ without allowing /wp-admin/admin-ajax.php can break features that depend on it. Allow the exception in the Allow box.
Putting the file in the wrong place. robots.txt only works at the site root, at yourdomain.com/robots.txt. In a subfolder, crawlers never find it and your rules have no effect.
Relying on robots.txt to hide content. Blocked pages can still appear in search results if other sites link to them, and the file is public. For content that must stay private, use a password, not a Disallow rule.
Related Tools You May Need Next
XML Sitemap Generator. Build the sitemap whose address goes in the Sitemap: line of your robots.txt.
Meta Tag Generator. Create the title and description tags for the pages your robots.txt lets crawlers visit.
Schema Markup Generator. Add structured data to those pages so search engines understand them.
Slug Generator. Make clean, short URLs for the pages you want crawled.
SEO Report. Check a page’s overall SEO once your crawling rules are in place.
SERP Checker. See where a page ranks after crawlers have visited it.
Google Cache Checker. Check whether Google has a cached copy of a page, which shows it was crawled.

Frequently Asked Questions
How do I generate a robots.txt file?
Choose Allow all or Disallow all, type the paths to disallow and allow one per line, add your sitemap address if you have one, and click Generate robots.txt. Copy the result, save it as robots.txt, and upload it to your site’s root folder.
What is a robots.txt file?
A plain-text file at the root of a website, at yourdomain.com/robots.txt, that tells search engine crawlers which parts of the site they may or may not crawl.
Where do I put the robots.txt file?
At the root of your site, so it is reachable at yourdomain.com/robots.txt. Crawlers only look there; in any other folder the file is ignored and your rules have no effect.
Can I make different rules for Googlebot and other crawlers?
Not with this tool. It always writes one group starting with User-agent: *, which applies to all crawlers. For crawler-specific rules, add a second group by hand to the copied text before uploading.
Does the tool download a robots.txt file for me?
No. It shows the text with a Copy button. Paste it into a plain-text editor, save it as robots.txt, and upload it to your site root yourself.
What does an empty Disallow line mean?
Disallow: with nothing after it means “disallow nothing”, so all crawlers may visit all pages. That is the output you get when you pick Allow all and add no paths.
Can robots.txt block content from search engines?
It can ask well-behaved crawlers not to visit parts of your site, and the major search engines respect it. But it is a request, not a lock: it does not stop anyone opening a page directly, and a blocked page can still appear in results if other sites link to it. Do not use it to protect private content.
What is a common robots.txt mistake?
Blocking too much — usually a stray Disallow: /, which hides the entire site from search results, or a path typed without its leading slash, which blocks nothing. Read the generated file before you upload it.
Is the robots.txt generator free?
Yes. It is free with no account and no sign-up, and it runs entirely in your browser, so there is nothing to install and no submission to count. Generate as many files as you like, copy the result, and upload it to your site.