SEO is the key to a successful online business nowadays. That is why it is really important to configure the robots.txt file properly. This file is a set of instructions given to the web crawlers “guiding” them which pages of your website you want and don't want to be indexed and analyzed.

What is robots.txt file?

Robots.txt. file is the most basic kind of SEO configuration available in Magento by default. It is essential to configure the robots.txt file in order to establish kind of a relationship between your website robots and website crawlers.

Configuring the robots.txt file you set the instructions to web robots on what pages of your website to index or skip from indexing.

How to configure robots.txt file in Magento 2?

You can use the default Magento settings. However, in case there are some pages of your website used internally, which you want to "hide" from search crawlers, you can add your custom configurations to robots.txt. file.

To configure robotx.txt file in Magento 2:

1. Navigate to Content > Design > Configuration.

Magento 2 Design Configuration

2. Choose the Store you want to configure the default robots.txt file for.
Magento 2 design configuration store

3. Find the Search Engine Robots section.

4. Choose the Default Robots you want to be enabled on your website.

There are several options you can choose from. Let's see what every default robot option stands for:

  • If you want web crawlers to index a page and follow the links on that page - INDEX, FOLLOW
  • If you don’t want web crawlers to index a page but want them to follow the links on that page - NOINDEX, FOLLOW
  • If you want web crawlers to index a page and don't want them to follow the links on that page - INDEX, NOFOLLOW
  • If you want web crawlers neither to index a page nor to follow the links on that page - NOINDEX, NOFOLLOW

5. In the Edit custom instruction of the robot.txt File section enter the custom instruction you want the search engine to analyze your website according to. Check the examples of the custom robots.txt instructions later in this article.

6. Click the Reset To Defaults button in case you want to delete all your custom instructions and go back to the default ones.

7. After you set the default robots, don't forget to Save Configuration.

Magento 2 search engine robots

Once you saved the configuration, you may be asked to flush the cache.

Magento robots.txt example

To "hide" some pages of your website from the web crawlers you can set custom instructions in the robots.txt file. Here some examples:

  • Allow Full Access
User-agent:*
Disallow:
  • Disallow Access to All Folders
User-agent:*
Disallow: /
  • Default Instructions
Disallow: /lib/
Disallow: /*.php$
Disallow: /pkginfo/
Disallow: /report/
Disallow: /var/
Disallow: /catalog/
Disallow: /customer/
Disallow: /sendfriend/
Disallow: /review/
Disallow: /*SID=
  • Restrict Checkout and Customer account

Disallow: /checkout/
Disallow: /onestepcheckout/
Disallow: /customer/
Disallow: /customer/account/
Disallow: /customer/account/login
  • Restrict Catalog Search Pages

Disallow: /catalogsearch/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/
Disallow URL Filter Searches
Disallow: /*?dir*
Disallow: /*?dir=desc
Disallow: /*?dir=asc
Disallow: /*?limit=all
Disallow: /?mode
  • Restrict Common files
Disallow: /composer.json
Disallow: /composer.lock
Disallow: /CONTRIBUTING.md
Disallow: /CONTRIBUTOR_LICENSE_AGREEMENT.html
Disallow: /COPYING.txt
Disallow: /Gruntfile.js
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt
Disallow: /nginx.conf.sample
Disallow: /package.json
Disallow: /php.ini.sample
Disallow: /RELEASE_NOTES.txt
  • Restrict Common Folders
Disallow: /app/
Disallow: /bin/
Disallow: /dev/
Disallow: /lib/
Disallow: /phpserver/
Disallow: /pub/
  • Restrict Technical Magento Files
Disallow: / api.php
Disallow: / cron.php
Disallow: / cron.sh
Disallow: / error_log
Disallow: / get.php
Disallow: / install.php
Disallow: / LICENSE.html
Disallow: / LICENSE.txt
Disallow: / LICENSE_AFL.txt
Disallow: / README.txt
Disallow: / RELEASE_NOTES.txt
  • Restrict CMS Directories
Disallow: /app/
Disallow: /bin/
Disallow: /dev/
Disallow: /lib/
Disallow: /phpserver/
Disallow: /pub/

How to add a sitemap to robots.txt file in Magento 2?

Sitemaps play a really important role in your website SEO since they allow search engines to analyze your website links better. And since the robots.txt file is a set of instructions for what to analyze, it is a good practice to add a sitemap to the robots.txt file.

In order to add the sitemap to the robots.txt file take these steps:

1. Navigate to Store > Configuration > Catalog > XML Sitemap and find the Search Engine Submission Settings section.

2. Enable Submission to Robots.txt option.

Magento 2 XML sitemap configuration

In case you want this option to be enabled automatically, navigate to Content > Design > Configuration > Search Engine Robots and in the Edit custom instruction of the robot.txt File field set the instructions to add the sitemap as in the example:

Sitemap: https://magefan.com/pub/sitemaps/sitemap.xml
Sitemap: https://magefan.com/pub/sitemaps/blog_sitemap.xml
Sitemap: https://magefan.com/pub/sitemaps/sitemap_ua.xml
Sitemap: https://magefan.com/pub/sitemaps/blog_sitemap_ua.xml

If your website is under development, you might want to restrict its indexing by web robots. In order to do that you have to choose NOINDEX, NOFOLLOW in the Default Robots field. (Content > Design > Configuration > Search Engine Robots).

In case you already have a set of instructions for the web robots to analyze your website, but haven't configured your website sitemap yet, check this article on Magento 2 Sitemap Configuration.

You will learn not only how to add the XML sitemap in Magento, but how to configure it and how to set Magento 2 sitemap auto-generation. So, go ahead and check it out!