Stand with SEO is the key to a successful online business nowadays. That is why it is really important to configure the robots.txt file properly. This file is a set of instructions given to the web crawlers “guiding” them which pages of your website you want to be crawled and indexed.
It mainly consists of the INDEX and FOLLOW tags. At the same time, NOINDEX NOFOLLOW tags in Magento are used to remove some pages from being discovered by search crawlers.
So, in this article, you will learn what is Magento 2 robots.txt, where is this file located and, most importantly, how to generate it.
Let's dive in.
Post Contents [hide]
What is Magento 2 robots.txt file?
Robots.txt. file is the most basic kind of Magento SEO configuration available by default. It is essential to configure the Magento robots.txt file in order to establish kind of a relationship between your website robots and website crawlers.
Configuring the robots.txt file you set the instructions to web robots on what pages of your website to index or skip from indexing.
How to configure Magento 2 robots.txt file?
You can use the default Magento settings. However, in case there are some pages of your website used internally, which you want to "hide" from search crawlers, you can add your custom configurations to robots.txt. file.
To configure Magento 2 robots.txt:
1. Navigate to Content > Design > Configuration.
2. Choose the Website you want to configure the default Magento robots.txt file for. Please note that robots.txt option is not available on a store view level, only for websites or global.
3. Find the Search Engine Robots section.
4. Choose the Default Robots you want to be enabled on your website.
There are several options you can choose from. Let's see what every default robot option stands for in Magento 2 robots.txt:
- If you want web crawlers to index a page and follow the links on that page - INDEX, FOLLOW
- If you don’t want web crawlers to index a page but want them to follow the links on that page - NOINDEX, FOLLOW
- If you want web crawlers to index a page and don't want them to follow the links on that page - INDEX, NOFOLLOW
- If you want web crawlers neither to index a page nor to follow the links on that page - NOINDEX, NOFOLLOW
5. In the Edit custom instruction of the robot.txt File section enter the custom instruction you want the search engine to analyze your website according to. Check the examples of the custom robots.txt instructions later in this article.
6. Click the Reset To Defaults button in case you want to delete all your custom instructions and go back to the default ones.
7. After you set the default robots, don't forget to Save Configuration.
Once you saved the configuration, you may be asked to flush the cache.
Magento 2 robots.txt sample
User-agent: * User-agent: Googlebot User-agent: Googlebot-image # Paths (clean URLs) Disallow: /*? Allow: /*?page= Allow: /*?p= Allow: /graphql? Disallow: /*SID= Disallow: /repo/ Disallow: /catalog/product_compare/ Disallow: /catalog/category/view/ Disallow: /catalog/product/view/ Disallow: /catalog/seo_sitemap/ Disallow: /catalogsearch/ Disallow: /mfproductsearch/ Disallow: /checkout/ Disallow: /control/ Disallow: /customer/ Disallow: /customize/ Disallow: /sendfriend/ Disallow: /ajaxcart/ Disallow: /ajax/ Disallow: /quickview/ Disallow: /productalert/ Disallow: /mfcmsdr/ Disallow: /sales/guest/form/ Disallow: /sales/guest/form/ Disallow: /review/ Disallow: /downloadable/ Disallow: /pslogin Disallow: /subscription Disallow: /newsletter Disallow: /push_notification # Files Disallow: /index.php Disallow: /cron.php Disallow: /cron.sh Disallow: /error_log Disallow: /install.php Disallow: /LICENSE.html Disallow: /LICENSE.txt Disallow: /LICENSE_AFL.txt Disallow: /STATUS.txt Disallow: /get.php Disallow: /app/ Disallow: /lib/ Disallow: /*.php$ Disallow: /pkginfo/ Disallow: /report/ Disallow: /var/ #CMS Pages Disallow: /privacy-policy-cookie-restriction-mode Disallow: /no-route Disallow: /enable-cookies Disallow: /home
How to add sitemap to robots.txt file in Magento 2?
Sitemaps play a really important role in your website SEO since they allow search engines to analyze your website links better. And since the robots.txt file is a set of instructions for what to analyze, it is a good practice to add a sitemap to this file.
In order to add the sitemap to the robots.txt file take these steps:
1. Navigate to Store > Configuration > Catalog > XML Sitemap and find the Search Engine Submission Settings section.
2. Enable Submission to Robots.txt option.
In case you want this option to be enabled automatically, navigate to Content > Design > Configuration > Search Engine Robots and in the Edit custom instruction of the robot.txt File field set the instructions to add the sitemap as in the example:
Sitemap: https://magefan.com/pub/sitemaps/sitemap.xml Sitemap: https://magefan.com/pub/sitemaps/blog_sitemap.xml Sitemap: https://magefan.com/pub/sitemaps/sitemap_ua.xml Sitemap: https://magefan.com/pub/sitemaps/blog_sitemap_ua.xml
If your website is under development, you might want to restrict its indexing by web robots. In order to do that you have to choose NOINDEX, NOFOLLOW in the Default Robots field. (Content > Design > Configuration > Search Engine Robots).
In case you already have a set of instructions for the web robots to analyze your website but haven't configured your website sitemap yet, check this article on Magento 2 Sitemap Configuration.
You will learn not only how to add the XML sitemap in Magento, but how to configure it and how to set Magento 2 sitemap auto-generation. So, go ahead and check it out!
