SEO is the key to a successful online business nowadays. That is why it is really important to configure the robots.txt file properly. This file is a set of instructions given to the web crawlers “guiding” them which pages of your website you want and don't want to be indexed and analyzed.

So, in this article, you will learn what is Magento robots.txt, where is this file located and, most importantly, how to generate it.

Let's dive in

Post Contents [hide]

    What is Magento 2 robots.txt file?

    Robots.txt. file is the most basic kind of SEO configuration available in Magento by default. It is essential to configure the robots.txt file in order to establish kind of a relationship between your website robots and website crawlers.

    Configuring the robots.txt file you set the instructions to web robots on what pages of your website to index or skip from indexing.

    How to configure robots.txt file in Magento 2?

    You can use the default Magento settings. However, in case there are some pages of your website used internally, which you want to "hide" from search crawlers, you can add your custom configurations to robots.txt. file.

    To configure robotx.txt file in Magento 2:

    1. Navigate to Content > Design > Configuration.

    Magento 2 Design Configuration

    2. Choose the Store you want to configure the default robots.txt file for.
    Magento 2 design configuration store

    3. Find the Search Engine Robots section.

    4. Choose the Default Robots you want to be enabled on your website.

    There are several options you can choose from. Let's see what every default robot option stands for:

    • If you want web crawlers to index a page and follow the links on that page - INDEX, FOLLOW
    • If you don’t want web crawlers to index a page but want them to follow the links on that page - NOINDEX, FOLLOW
    • If you want web crawlers to index a page and don't want them to follow the links on that page - INDEX, NOFOLLOW
    • If you want web crawlers neither to index a page nor to follow the links on that page - NOINDEX, NOFOLLOW

    5. In the Edit custom instruction of the robot.txt File section enter the custom instruction you want the search engine to analyze your website according to. Check the examples of the custom robots.txt instructions later in this article.

    6. Click the Reset To Defaults button in case you want to delete all your custom instructions and go back to the default ones.

    7. After you set the default robots, don't forget to Save Configuration.

    Magento 2 search engine robots

    Once you saved the configuration, you may be asked to flush the cache.

    Magento 2 robots.txt example

    To "hide" some pages of your website from the web crawlers you can set custom instructions in the robots.txt file. Here some examples:

    • Allow Full Access
    User-agent:*
    Disallow:
    • Disallow Access to All Folders
    User-agent:*
    Disallow: /
    • Default Instructions
    Disallow: /lib/
    Disallow: /*.php$
    Disallow: /pkginfo/
    Disallow: /report/
    Disallow: /var/
    Disallow: /catalog/
    Disallow: /customer/
    Disallow: /sendfriend/
    Disallow: /review/
    Disallow: /*SID=
    • Restrict Checkout and Customer account

    Disallow: /checkout/
    Disallow: /onestepcheckout/
    Disallow: /customer/
    Disallow: /customer/account/
    Disallow: /customer/account/login
    • Restrict Catalog Search Pages

    Disallow: /catalogsearch/
    Disallow: /catalog/product_compare/
    Disallow: /catalog/category/view/
    Disallow: /catalog/product/view/
    Disallow URL Filter Searches
    Disallow: /*?dir*
    Disallow: /*?dir=desc
    Disallow: /*?dir=asc
    Disallow: /*?limit=all
    Disallow: /?mode
    • Restrict Common files
    Disallow: /composer.json
    Disallow: /composer.lock
    Disallow: /CONTRIBUTING.md
    Disallow: /CONTRIBUTOR_LICENSE_AGREEMENT.html
    Disallow: /COPYING.txt
    Disallow: /Gruntfile.js
    Disallow: /LICENSE.txt
    Disallow: /LICENSE_AFL.txt
    Disallow: /nginx.conf.sample
    Disallow: /package.json
    Disallow: /php.ini.sample
    Disallow: /RELEASE_NOTES.txt
    • Restrict Common Folders
    Disallow: /app/
    Disallow: /bin/
    Disallow: /dev/
    Disallow: /lib/
    Disallow: /phpserver/
    Disallow: /pub/
    • Restrict Technical Magento Files
    Disallow: / api.php
    Disallow: / cron.php
    Disallow: / cron.sh
    Disallow: / error_log
    Disallow: / get.php
    Disallow: / install.php
    Disallow: / LICENSE.html
    Disallow: / LICENSE.txt
    Disallow: / LICENSE_AFL.txt
    Disallow: / README.txt
    Disallow: / RELEASE_NOTES.txt
    • Restrict CMS Directories
    Disallow: /app/
    Disallow: /bin/
    Disallow: /dev/
    Disallow: /lib/
    Disallow: /phpserver/
    Disallow: /pub/

    How to add sitemap to robots.txt file in Magento 2?

    Sitemaps play a really important role in your website SEO since they allow search engines to analyze your website links better. And since the robots.txt file is a set of instructions for what to analyze, it is a good practice to add a sitemap to this file.

    In order to add the sitemap to the robots.txt file take these steps:

    1. Navigate to Store > Configuration > Catalog > XML Sitemap and find the Search Engine Submission Settings section.

    2. Enable Submission to Robots.txt option.

    Magento 2 XML sitemap configuration

    In case you want this option to be enabled automatically, navigate to Content > Design > Configuration > Search Engine Robots and in the Edit custom instruction of the robot.txt File field set the instructions to add the sitemap as in the example:

    Sitemap: https://magefan.com/pub/sitemaps/sitemap.xml
    Sitemap: https://magefan.com/pub/sitemaps/blog_sitemap.xml
    Sitemap: https://magefan.com/pub/sitemaps/sitemap_ua.xml
    Sitemap: https://magefan.com/pub/sitemaps/blog_sitemap_ua.xml
    

    If your website is under development, you might want to restrict its indexing by web robots. In order to do that you have to choose NOINDEX, NOFOLLOW in the Default Robots field. (Content > Design > Configuration > Search Engine Robots).

    In case you already have a set of instructions for the web robots to analyze your website, but haven't configured your website sitemap yet, check this article on Magento 2 Sitemap Configuration.

    You will learn not only how to add the XML sitemap in Magento, but how to configure it and how to set Magento 2 sitemap auto-generation. So, go ahead and check it out!