Here’s the best robots.txt file for your Blogger site, ensuring no redirect errors while following best SEO practices:
Optimized robots.txt for Blogger (No Redirect Errors)
User-agent: *
Disallow: /search
Disallow: /label/
Disallow: /archive/
Disallow: /*.html$
Allow: /*.html?m=1$
Allow: /*?m=1$
Sitemap: https://ABC.blogspot.com/sitemap.xml
Breakdown of the Rules:
1. User-agent: *
This applies to all web crawlers, as the asterisk (*) is a wildcard.
2. Disallow Rules
Disallow: /search → Prevents bots from accessing URLs that start with /search.
Disallow: /label/ → Blocks access to URLs that start with /label/.
Disallow: /archive/ → Prevents bots from crawling pages under /archive/.
Disallow: /*.html$ → Blocks all pages that end in .html.
3. Allow Rules
Allow: /*.html?m=1$ → Allows pages ending in .html only if they include ?m=1 (which is often a mobile-friendly version).
Allow: /*?m=1$ → Allows any URL with ?m=1, ensuring mobile versions of pages can still be crawled.
Purpose:
- This setup is likely for a Blogger (Blogspot) website, as Blogger uses ?m=1 for mobile versions.
- It prevents unnecessary crawling of search results, labels (categories), and archives to optimize indexing.
- Ensures mobile-friendly pages remain accessible to search engines.
Why This is the Best Setup?
✅ No Redirect Errors – Ensures proper crawling without indexing unnecessary or duplicate pages.
✅ Better SEO – Prevents Google from indexing label, archive, and .html pages (avoiding duplicate content issues).
✅ Mobile-Friendly – Allows m=1 versions of posts and pages to be indexed correctly.
✅ Faster Indexing – Sitemap helps search engines discover and index important pages efficiently.
How to Add This in Blogger?
1. Go to Blogger Dashboard
2. Go to Settings > Crawlers and Indexing
3. Enable "Custom robots.txt"
4. Paste the above robots.txt code
5. Save Changes
This setup ensures your Blogger site is fully optimized while avoiding indexing errors or duplicate content problems. Let me know if you need any adjustments!