🤖 SEO Tool

Robots.txt Generator & Tester

Build a properly formatted robots.txt file using a visual form. Add rules for multiple bots, set allowed and blocked paths, include your sitemap, and download the file ready to upload.

Free Forever Visual Builder URL Tester Built-in Instant Download

⚡ Quick Presets

🗺️ Sitemap URL

Sitemap URL (highly recommended)

Adding your sitemap URL helps Google find all your pages faster.

📋 Bot Rules

⏱️ Global Crawl Delay (optional)

Crawl Delay (seconds)

Not supported by Googlebot, but respected by some other crawlers like Bing.

📄 robots.txt Output

# robots.txt generated by ToolHub
# toolhub.io — Free SEO & Dev Tools

User-agent: *
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

🧪 URL Tester — Check if a URL is Blocked

Enter a URL path to see if your current robots.txt rules would allow or block it.

💡 Upload Instructions

Click Download to save as robots.txt
Upload the file to your website root directory (e.g. public_html/)
It must be accessible at yoursite.com/robots.txt
Test it using Google Search Console

📖 About This Tool

What is robots.txt?

The robots.txt file is a plain text file placed at the root of your website that instructs web crawlers which pages or sections they are allowed to visit. It uses the Robots Exclusion Protocol — a simple set of directives understood by all major search engine bots.

A well-configured robots.txt keeps crawl budget focused on your important pages, prevents duplicate content issues, protects admin areas from being indexed, and helps search engines understand your site structure more efficiently.

⚠️

Not a Security Tool

robots.txt prevents indexing but does not block access. Sensitive content should be protected with proper authentication — not just robots.txt directives.

🤖

Googlebot vs Others

Googlebot respects robots.txt reliably. Malicious bots often ignore it entirely. Use it for SEO crawl management, not as a firewall.

📏

500KB Limit

Google only processes the first 500KB of a robots.txt file. Keep it concise. Group related rules together and use wildcards to reduce file size.