Robots.txt Generator

Robots.txt is a small plain text file in the root of your website that tells search engine crawlers which paths they can fetch and which they cannot. It is the first file most bots request before they crawl, and it sits at the same address as your homepage with the path slash robots dot txt.

A bad robots.txt can quietly block your entire site from Google. A good one keeps crawlers focused on the pages you want indexed and away from low value or private sections like admin panels, cart pages, and internal search results. It also points crawlers to your sitemap.

1. Decide if you want to allow or block all bots by default.
2. List paths to disallow, one per line.
3. Optionally add a crawl delay and your sitemap URL.
4. Click Generate, then upload the file as robots.txt to the root of your site.

Never block JavaScript and CSS files. Google needs them to render the page like a real browser. Blocking them used to be common advice that now hurts rankings.