If you want your pages to be crawled and indexed accordingly, you need to provide instructions for search robots in the robots.txt file. However, it works both ways.
When you need to hide certain pages from search permanently or for a while you need to set NOINDEX NOFOLLOW or other tags for those pages. You can do that using the robot.txt or HTML head of the pages.
In this guide, you'll learn an easy non-technical way to apply custom robot meta tags to one or multiple pages in bulk.
There are two ways how you can set custom robot meta tags in Magento — for one or multiple pages at once. We'll go through each of them in detail.
Set NOINDEX NOFOLLOW tags for one page
Each Magento page contains the Design section where you can add custom robot meta tags. So, to add a NOINDEX NOFOLLOW tag for any of them you should go to the page you'd like to hide from robots, scroll down to the Design section and find the Layout Update XML field.
Add the following code to it:
<head>
<meta name="robots" content="NOINDEX,NOFOLLOW"/>
</head>
Don't forget to Save the page and check the robots on the frontend. Go to the page and use the Ctrl+U combination or Inspect browser tools to find the robots.
This method is relatively simple since all you need is to paste a small snippet of code into the Design section. But it's not that effective if you have multiple pages to hide from meta robots. So, jump to the next section to find a more effective solution.
Set NOINDEX NOFOLLOW tags for multiple pages
When you need to add custom robot meta tags for multiple pages it's either for products or categories. Adding robots meta tags to each page can be tiresome...
... unless you use the Magento SEO extension to create SEO rules.
They allow you to set NOINDEX NOFOLLOW tags for multiple product or category pages at once. Let's start with products.
To set NOINDEX NOFOLLOW meta robots for products in Magento 2:
1. Navigate to Marketing > SEO by Magefan > SEO Rules and click on the Add New Rule button.
2. Enable a rule, set a Name and specify a Priority.
3. Choose Product as a Page Type.
4. Specify Meta Robots you'd like to add for certain product pages.
Note: H1 Title, Description, Short Description, etc. are optional when you set custom meta robots. They are used to create SEO rules in Magento.
5. Select what Store View to set the robots for.
6. Set the Conditions using product attributes to determine what products to apply custom robots to. You can also use the Preview option to check what products the robots will be applied to.
In case you want to set custom meta tags for categories:
- Choose Category as a Page Type.
- Set custom Meta Robots.
- Choose whether to Use Rule for Child Categories and choose Categories which you want to apply custom meta tags for.
Don't forget to Save the rule and press the Apply Rules button on the rules grid if you want to apply it immediately.
Pro tip: if you want to set NOINDEX NOFOLLOW tags for products or categories permanently, it's best to exclude those pages from the XML sitemap altogether.
When you have content that you don't want to be visible in search, custom robot meta tags are exactly what you need. But that's just a small part of SEO have to cover to optimize your store.
