Getting Google to discover your blog pages is easy. You just need to upload your sitemap to Google Search Console and make sure blog pages are not blocked in the robots.txt file. But that's not quite it.
You also have to make sure crawlers can index and display these pages in search engines. That's what robots meta tags are for.
So, today you'll learn how to set robot meta tags for your blog using the extension. We won't stop on the directives here, so explore them in our Magento meta robots guide.
Default Robots Meta Tags
Navigate to Stores > Configuration > Magefan Extensions > Blog to find default meta robots settings for the Author's page.
The same option is available for Tag and Search pages here.
Once you set the default robots for the blog tag, author, and search pages, they will be included before the head closing tag in the page HTML.
Note: use the Ctrl+U combination or Inspect browser tools to check the blog robots meta tags.
The only drawback of the default blog meta robot settings is that they are applied for all pages in bulk. But we've got you covered.
Robots Meta Tags for Specific Pages
If you want to change robots meta tags for some specific page you can do that directly on that page. Just navigate to the blog posts, category, tag or author page (Content > Blog > Posts/Categories/Tags/ Authors) and scroll down to the Search Engine Optimization section.
Each blog page has a separate Meta Robots option there.
Meta Robots vs Blog Sitemap
Once you set custom robots meta tags for any blog page, web robots will crawl and index it accordingly. But there is also one important thing to remember here.
Pro tip: never include blog pages with NOINDEX tags in the blog sitemap. This sends confusing signals to the search crawlers and might cause indexing issues.
So, once you add NOINDEX robots to the blog pages, make sure to exclude these pages from the blog sitemap too. Magento XML Sitemap Extension might help with this.
Blog robots meta tags are part of the bigger Magento SEO picture of your store. While we've optimized them according to the best practices by default, make sure you follow them too when changing the robots.
