Build a robots.txt without the guesswork
Choose whether to allow or block crawlers by default, list the paths you want to keep out of search, add a crawl-delay or your sitemap, and the tool assembles a correctly formatted robots.txt you can copy straight to your site's root. No syntax to memorize.
What each directive does
- User-agent — which crawler the rules apply to; * means all of them.
- Disallow — a path prefix crawlers should not request, like /admin/.
- Allow — an exception that permits a path inside a disallowed area.
- Sitemap — the full URL of your sitemap so crawlers can find all your pages.
Important limits
Save the output as a plain text file named robots.txt and upload it to your domain root, e.g. https://example.com/robots.txt.
Frequently asked questions
What is a robots.txt file?
A robots.txt file lives at the root of your domain and tells search engine crawlers which parts of your site they may or may not request. It's a set of guidelines crawlers follow, placed at yourdomain.com/robots.txt.
Does robots.txt keep a page out of Google?
Not reliably. Disallowing a URL stops compliant crawlers from fetching it, but a blocked page can still be indexed if it's linked elsewhere. To keep a page out of results, use a noindex meta tag or an authentication wall instead.
Where do I put the robots.txt file?
It must be at the root of your domain — for example https://example.com/robots.txt. A robots.txt in a subfolder is ignored. Save the generated text as a plain file named robots.txt and upload it there.