Build a proper robots.txt file for your website. Control which pages Google and other search engine crawlers can access and index. Download-ready output.
π€ All Major Bots
β Validates Output
β¬ Download File
π No Sign-Up
π€ Configure your robots.txt
Fill in the options below to generate your file
β
All Bots (*)
β
Googlebot
β
Bingbot
β
Google Images
β
Google Ads
β
GPTBot (OpenAI)
β
/wp-admin/
β
/wp-login.php
β
/wp-includes/
β
/wp-content/plugins/
β
/?s= (search pages)
β
/feed/
π robots.txt Output
# Your robots.txt will appear here
# Fill in the form on the left and click Generate
β File Validation
βͺFill in the form to validate your robots.txt
π‘ Common Directives
Disallow: /
Block entire site from all crawlers
Allow: /
Allow entire site (default behaviour)
Disallow: /private/
Block a specific folder
Disallow: /*.pdf$
Block all PDF files
Disallow: /*?*
Block all URLs with query strings
π‘ Key Tips
π
robots.txt must be placed at the root: yoursite.com/robots.txt
β οΈ
robots.txt blocks crawling but NOT indexing β use noindex meta tags to prevent indexing.
π
Always include your sitemap URL in robots.txt β it helps Google discover your pages faster.
π€
GPTBot and CCBot scrape your content for AI training β add them to block AI crawlers.