WordPress Robots.txt Sample

WordPress Robots.txt User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ Adding Sitemaps to WordPress Robots.txt User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ Sitemap: http://www.example.com/post-sitemap.xml Explanation Allowing all Bots Allowing any Bots to Crawl User-agent: * Disallow: Not Allowing any Bots Not Allowing any Bots to Crawl User-agent: * Disallow: / Block a Folder User-agent: * Disallow: /Folder/ Block a File User-agent: … Continued