Getting Google to crawl and index your pages the way you want is quite challenging. That's because people often confuse credibility and indexation, which are roughly not the same thing. Robots.txt and sitemap might already be too much to handle in the technical part of Magento SEO. Then there are robots meta tags.
We'll get all of these straight in this guide. You'll learn what are robots meta tags in Magento, why they are important for SEO and how you can manage them properly.
Let's get right to it.
Post Contents [hide]
What are Robots Meta Tags?
Robots meta tags in an HTML code that tell search engine robots how to create, index and display the page content in search results. In other words, it tells web crawlers what they can and can't do on a certain page.
The mata tags robots are added in the <head> section of the pages and looks like this:
Since robots meta tags control how Google crawls and indexes your page, you need them to control:
- whether a page appears in search results
- whether images on the page are indexed
- whether the robots should follow the links on the page
- whether a snippet of the page is displayed in SERP
Robots meta tags are usually used to remove pages from Google indexing:
e.g. staging environment pages, admin pages, PPC landing pages, confirmation pages (checkout success), pages with duplicate content etc.
At this point, you might be wondering why you need these meta robots if you've already disallowed these pages in robots.txt. That's when things start to get interesting, so keep reading.
Meta Robots vs Robots.txt
The robots.txt and robots meta tags in Magento have similar functions. That's true. But that doesn't mean they are the same.
The robots.txt file contains instructions for the entire website and tells search crawlers what pages to crawl or skip.
The robots meta tags contain instructions only for the page containing the robots meta tag. It tells search crawlers how to crawl, index and display information from a specific page.
To put it simply, robot meta tags provide more firm instructions for indexation behaviour than robots.txt. In fact, with the latter search crawlers don't have to follow your meta directives. This leads to some malicious web robots even ignoring your directives.
Meta Robots Directives
Robots meta tags in Magento, as in any other environment, contain two attributes: name and content. While the name defines what crawler should follow the instructions in the tag, content contains instructions for that crawlers.
You'll most often stumble upon a robots metatag with the "robots" as the name:
meta name="robots"
It means that you address all web crawlers. However, you can also go for specific ones for Google, Bind, DuckDuckGo and Baidu.
Note: if you don't set any meta tags for robots, they will index and follow by default. This means they will show a page in search results and follow the links on the page.
While you mostly won't need to go beyond the noindex nofollow instructions, it's best to know about different options. So, let's explore.
Noindex
Tells search crawlers not to index and show a page in the search results.
<meta name="robots" content="noindex">
Nofollow
Instructs search robots not to crawl the links on the page (and pass links equally).
<meta name="robots" content="nofollow">
Note: the links on the page will still be indexable, especially if they have backlinks from other pages.
Noarchive
Tells Google not to show a cached copy of the page in search results.
<meta name="robots" content="noarchive">
Notranslate
Prevents Google from showing translations for the page in SERPs.
<meta name="robots" content="notranslate">
Noimageindex
Tells Google not to index images available on a page.
<meta name="robots" content="noimageindex">
Nosnippet
Instructs Google not to show a text or video preview of the page in search results.
<meta name="robots" content="nosnippet">
Max-snippet
Tells Google about the maximum number of characters it can show as a text snippet for the page. Using -1 sets no limit on the text preview characters, while 0 opts a page out of the snippets.
<meta name="robots" content="max-snippet:-1">
Max-image-preview
Instructs Google on how to use images for image snippets. The none indicates you don't want to show any images, standard — specifies to use default image preview, and large — tells you to use the largest possible image preview.
<meta name="robots" content="max-snippet:100">
Max-video-preview
Instructs Google on the number of seconds to use for a video snippet. The logic is the same as with the text snippets: -1 sets no limits, 0 opts a page out completely.
<meta name="robots" content="max-snippet:0">
Note: you can use multiple directives at the same time and combine them. But note that if they conflict with each other, Google will use the more restrictive one e.g. "noindex, index"
How to Set and Manage Robots Meta Tags in Magento?
By default, Magento 2 adds INDEX and FOLLOW tags for all pages of your store that are not listed in the robots.txt file. However, as you already know, it still might not be enough to stop web crawlers from indexing specific pages.
So, we'll teach you how you can set meta tags robots in Magento 2.
Robots.txt
Regardless of the fact that robots.txt alone is not enough to manage indexation, you need to set custom instructions there.
For that go to Content > Design > Configuration and choose a website you want to set custom robots.txt instructions for. In the Search Engine Robots section, you can disallow indexing of certain pages — Edit custom instruction of the robot.txt File text area. Besides, you can also edit Default Robots here.
Design Section
The other way around is the Design section on each page of your website. You can set custom meta robots there. Go to a page you want to set robots meta tags for (Content > Pages) and scroll down to the Design section.
Then add the following code to the Layout Update XML field, specifying your custom directions for the robots.
<head>
<meta name="robots" content="NOINDEX,NOFOLLOW"/>
</head>
SEO Rules
If you don't want to spend countless hours configuring custom robots meta tags in Magento, there is a solution — SEO Rules. They allow you to set Magento meta robots for some specific pages or multiple pages in bulk.
Simply go to Marketing > SEO by Magefan > SEO Rules and create a new rule. Once you fill out general rule information, you need to choose what page to create rules for and specify the Meta Robots.
To add robots meta tags to multiple Magento pages, set conditions based on product attributes:
Robots Meta Tags Best Practices
Considering the number of different features that influence the crawlability and indexation of your pages, things might get confusing pretty soon. That's why you should follow the best practices when you set meta robots for your pages.
Don't add robots to pages blocked in robots.txt
If you disallow indexing of a new page in robots.txt it won't be indexed. It should be enough to prevent it from appearing in search results.
However, if the page has been crawled and indexed for a while, it's best to keep it "allowed" in the robots.txt. Instead, prevent it from indexing by custom meta robots.
Don't remove "noindex" pages from website sitemap
Don't rush to exclude a page from Megento sitemap, if you want to remove it from indexing. Add the "noindex" tag for it, but make sure it is deindexed first. Then remove it from the sitemap.
Remove the "noindex" directive when moving to production
Admin pages, dev and stage environment are not supposed to be in organic results. So it is a good practice to add "noindex" directives for the staging environment. Up until the moment when you move to production.
When you push your website to production and forget to change the robots, chances are you'll face some crawling and indexing issues (not necessarily from the start).
What's Next?
Robots meta tags in Magento help you to get your pages crawled, indexed and ranked in search results and vice versa. However, they don't guarantee lots of clicks. Once the page is in search, it's your job to make it "clickable" and attractive. For that, you need rich snippets and meta tags templates.
