JSONToTable
Loading Robots.txt Generator...
Please wait a moment

How to Generate a Robots.txt File

Control how search engine bots crawl and index your website with a properly formatted robots.txt file

Step 1

Configure Your Crawl Rules

Define which search engine bots can access your site and which pages or directories they should skip. The Robots Exclusion Protocol is a standard supported by Google, Bing, and all major search engines. You can add multiple rule groups for different user agents.

•
User-agent: The crawler to target — use * for all bots, or specify Googlebot, Bingbot, etc.
•
Disallow: Paths the crawler should NOT access (e.g. /admin/, /private/). Pair with a noindex meta tag for guaranteed exclusion from results.
•
Allow: Paths to explicitly allow even within a disallowed directory
•
Crawl-delay: Pause (in seconds) between each crawler request to reduce server load
Step 2

Generate and Download

The robots.txt output generates automatically as you configure your rules. Download it and upload it to the root of your website domain (e.g. https://example.com/robots.txt). For a complete SEO setup, pair your robots.txt with meta tags, Open Graph tags, and JSON-LD structured data.

Example robots.txt Output

User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /public/

User-agent: Googlebot
Crawl-delay: 1

Sitemap: https://example.com/sitemap.xml
Step 3

Upload to Your Website Root

The robots.txt file must be placed at the root of your domain. After uploading, verify it's accessible and test it in Google Search Console. Use the Google Robots Testing Tool to validate your rules before deploying.

•
Upload to your site root: https://yourdomain.com/robots.txt
•
Test in Google Search Console under Settings → Crawl Stats
•
Avoid blocking CSS, JS, or important pages that need to be crawled for indexing
•
Complete your SEO setup with JSON-LD structured data to unlock Google rich results

Frequently Asked Questions

What is a robots.txt file?

A robots.txt file tells search engine crawlers which pages or sections of your website they can or cannot access. It's placed at the root of your domain and is one of the first files crawlers check when visiting your site. Read Google's official robots.txt documentation for the full specification.

Does robots.txt prevent pages from being indexed?

Disallowing a page in robots.txt prevents crawlers from accessing it, but it doesn't guarantee the page won't appear in search results. If other sites link to a disallowed page, it can still be indexed. To prevent indexing, use a noindex robots directive — generate one with our Meta Tag Generator.

Where should I place my robots.txt file?

The robots.txt file must be placed at the root of your domain, accessible at https://yourdomain.com/robots.txt. It only applies to the domain or subdomain it's placed on — it does not apply to subdomains or other domains. Verify it's working in Google Search Console.

Should I add my sitemap to robots.txt?

Yes, adding your sitemap URL to robots.txt is a best practice. It helps search engines discover your sitemap quickly and ensures all your important pages get crawled and indexed efficiently.

What other SEO tools should I use alongside robots.txt?

For a complete SEO setup: use our Meta Tag Generator for page metadata, Open Graph Generator for social sharing previews, and JSON-LD Schema Generator to unlock Google rich results.

Related Tools

API Request Tester

Test REST APIs with all HTTP methods, headers, auth, and cURL import - like Postman

API Testing Tool

Test and debug REST APIs with comprehensive request and response analysis

Test API

Quick API endpoint testing with multiple HTTP methods and authentication

Compare Text

Compare and diff text files side by side

Open JSON File

Open JSON files online with drag & drop interface, upload and analyze JSON data instantly

JSON Reader

Read, parse and analyze JSON files online