Robots.txt Generator
FreeVisually build and preview your robots.txt file by selecting user-agents, crawl rules, and a sitemap URL. Copy the generated file and upload it to your site root to control crawl access instantly.
Generated robots.txt
User-agent: * Allow: /
How to use the Robots.txt Generator
- 1Select the user-agent you want to configure (all crawlers, Googlebot, Bingbot, etc.).
- 2Tick the crawl rules you want to apply — allow or disallow specific paths.
- 3Optionally enter your sitemap URL so search engines can find it.
- 4Review the live robots.txt preview.
- 5Click Copy and paste the content into a robots.txt file at your site root.
Related SEO Tools
Robots.txt Generator FAQs
What is a robots.txt file?
A robots.txt file is a plain text file placed at the root of your website (e.g. gocontento.io/robots.txt). It tells search engine crawlers which pages or sections of your site they are permitted to crawl and index.
Does disallowing a page in robots.txt remove it from search results?
Not necessarily. Disallowing a page blocks crawlers from visiting it, but if the page is already indexed or linked from other sites, it may still appear in results. Use a noindex meta tag to prevent indexing.
Can I have multiple user-agent sections?
Yes. A robots.txt file can contain multiple User-agent blocks, each with their own Allow/Disallow rules. Generate and combine multiple blocks using this tool for different crawlers.
Where should I put my robots.txt file?
Your robots.txt file must be placed at the root of your domain — for example, https://gocontento.io/robots.txt. It cannot be in a subdirectory and must be accessible without any redirects.
Need an SEO agency?
Browse verified SEO agencies and request proposals matched to your goals.