Build a proper robots.txt file for your website. Block AI scrapers, set WordPress presets, add custom rules, and get live validation — all with one-click download.
🤖 Block AI Bots
⚡ WordPress Presets
✅ Live Validation
⬇ Download File
10+
bot types supported
AI
GPTBot + CCBot blocking
WP
WordPress presets built-in
🤖 Configure your robots.txt
Fill in the options below — output updates automatically
✓
All Bots (*)
✓
Googlebot
✓
Bingbot
✓
Googlebot-Image
✓
AdsBot-Google
✓
Slurp (Yahoo)
✓
GPTBot (OpenAI)
✓
CCBot (Common Crawl)
✓
ClaudeBot (Anthropic)
✓
Google-Extended
✓
FacebookBot
✓
PerplexityBot
✓
/wp-admin/
✓
/wp-login.php
✓
/wp-includes/
✓
/wp-content/plugins/
✓
/?s= (search pages)
✓
/feed/
📄 robots.txt Output
# Your robots.txt will appear here
# Fill in the form on the left and click Generate
0 lines · 0 bytesrobots.txt
✅ File Validation
⚪Fill in the form to validate your robots.txt
💡 Common Directives
Disallow: /
Block entire site from all crawlers
Allow: /
Allow entire site (default behaviour)
Disallow: /private/
Block a specific folder
Disallow: /*.pdf$
Block all PDF files
Disallow: /*?*
Block all URLs with query strings
💡 Key Tips
📍
robots.txt must be placed at the root: yoursite.com/robots.txt — upload via Hostinger File Manager.
⚠️
robots.txt blocks crawling but NOT indexing. Use noindex meta tags to prevent indexing.
🔗
Always include your sitemap URL in robots.txt — it helps Google discover pages faster.
🤖
GPTBot, CCBot, and Google-Extended scrape content for AI training. Block them to protect your content.
🔄
After uploading, test at search.google.com/search-console/robots-testing-tool.