When visitors use layered navigation in Magento, it generates a lot of filtered URLs, but the bigger problem is when bots create thousands of filtered combinations by requesting those URLs repeatedly. Thus, Magento 2 layered navigation spam increases unnecessary requests, wastes your server resources and consumes the crawl budget.
According to the Magento 2 SEO guide you can control how filtered URLs are crawled and indexed through robots.txt, noindex tags and canonicals. But it doesn't stop bots from requesting and creating those URLs on your store.
This guide teaches you how to fix layered navigation spam in Magento, how to identify excessive filtering and how to restrict those requests to bots only, so real users can still use it.
What is Magento Layered Navigation Spam?
Magento layered navigation spam is automated bot traffic that repeatedly requests a large number of category URLs with multiple combinations or product attributes (filters), resulting in thousands of filtered URLs with little to no value.
This create two major problems for Magento stores:
- SEO issues — search robots will discover and crawl thousands of low-value parameterised URLs, which waste the crawl budget.
- Performance issues — each request that creates those parameterised URLs requires Magento to process it, which can consume too much of the server resources and lead to slow loading times and even time-outs.
Since layered navigation itself is not an issue, the goal is not to disable layered navigation, it's to identify abusive filtering behaviour coming from bots and block it.
How Bots Generate Thousands of Layered Navigation URLs?
Bots generate large numbers of layered navigation URLs by automatically combining multiple filter parameters already available on a Magento store, instead of browsing like legitimate users.
Even if your store has only a dozen on those filterable attributes, combining those values multiple times in different variations quickly grows the number of filtered URLs.
Bots usually create filtered pages in Magento in these ways:
- Following existing filter URLs: a bot can follow existing layered navigation links and continue applying filters to it.
- Requesting the same filter repeatedly: a bot can send multiple requests to the same filtered URL over a short period of time.
- Exploring unusual combinations: a bot may request filter combinations real users may never use.
- Creating new URL combinations: bots can create parameterised URLs by combining new filters instead of discovering existing filter URLs.
- Crawling pagination pages along layered navigation: bots may apply filters on top of the pagination pages that only multiplies the number of low-quality URLs.
To identify layered navigation spam in Magento, you should look for patterns rather than a suspicious filter combination. These could be a large volume of requests with layered navigation filters or many filter combinations requested over a short period of time, or unusual traffic spikes on category pages without any campaigns running.
Alternatively, you can use Magefan Bot Traffic Monitor to see what URL patterns are targeted the most and from what IPs and user agents.
Can Robots.txt and Noindex Fix Layered Navigation Spam in Magento?
Robots.txt and NOINDEX NOFOLLOW tags in Magento help to control crawling and indexing of the layered navigation URLs. But neither of these SEO features stops bots from requesting and generating those URLs repeatedly.
Proper bot-blocking rules are required to prevent bots from generating excessive numbers of filtered URLs in Magento.
Note: you can assign NOINDEX, NOFOLLOW tags to the layered navigation pages in bulk using the SEO Rules available in the Magefan's .
How to Fix Layered Navigation Spam in Magento?
To fix layered navigation spam in Magento, you need to configure a protection rule in Magefan's that detects excessive filtered requests and enables CAPTCHA challenges for suspicious visitors.
It allows you to define what qualifies as excessive filtering activity, so you can challenge anyone who exceeds that limit while keeping layered navigation enabled for everyone else.
First, navigate to Stores > Configuration > Magefan Extensions > Bot Blocker > Layered Navigation Protection and enable it.
Note: For layered navigation protection to work, you need to configure the reCAPTCHA challenges in the reCAPTCHA section of the extension.
If visitors exceed the limits you configure, they will be required to complete the CAPTCHA challenge.
Step 1: Set the minimum filter parameters
Before you configure the layered navigation limits, you shouldn't misunderstand how the extension counts filters.
e.g. in /shoes.html?color=red&size=10&brand=nike — color, size and brand are all counted as layered navigation filters, three in total.
The number of those filters is used in the First-Request Parameter Limit and Max Filtered Requests per Minute settings.
The Minimum Filter Parameters allows you to filter out normal requests and focus on suspicious combinations. It defines the number of parameters in the layered navigation URL that counts as spam.
e.g. if the Minimum Filter Parameters is set to 2, the requests containing one filter are not triggering the filters.
Set minimum filter parameters for layered navigation spam
Step 2: Set the layered navigation filter limits
The First-Request Parameter Limit detects visitors that land on your website with an unusual number of filters already applied to the URL they land on. Set the limit on the number of filters a user is allowed to set on their first request within the last two minutes.
Important: this setting only checks the visitor's initial request (the first requested URL). So, if a visitor visits any other pages and uses a number of filters that exceeds the limits set in the First-Request Parameter Limit field, it won't be challenged with reCAPTCHA.
Finally, set the Max Filtered Requests per Minute to set a limit on how many layered navigation requests a single IP can make within a minute before reCAPTCHA is triggered (according to the Minimum Filter Parameters option).
e.g. if a bot sends multiple requests like:
?color=red&size=10
?color=blue&size=10
?brand=nike&size=10
with the minimum filter parameters set to 2, it counts as spam and triggers the CAPTCHA challenge.
Maximum and first-request parameter limits for layered navigation spam blocking
Step 3: Define layered navigation URL patterns
Use regex or plain-text patterns to define what URLs should be checked for malicious bot activity. You can specify a certain category or search URLs where you detected unusual bot activity.
URL patterns checked for layered navigation spam blocking
Note: if you use a layered navigation extension to create SEO-friendly URLs, you don't need to set a path here if it's already matched in the Path Filter Segment Pattern setting.
Step 4: Exclude URLs and filter parameters
Specify the URLs that should never be checked for spam, regardless of how many filters the request contains, in the Excluded URL Patterns section. This allows you to exclude requests that are part of normal layered navigation activity.
e.g. you can exclude internal URLs like
|/customer/section/load|
|/cache_warmer/|
|/customer/ajax/|
|/blockajaxloader/|
|/page_cache/block/render/|
URLs excluded from the layered navigation spam blocker
Then specify the parameters that shouldn't be counted as layered navigation filters in the Excluded Query Parameters field. This setting helps you exclude sorting, tracking, pagination, store switching and other filters that are not classified as layered navigation filters.
e.g. cat,utm_source,utm_medium,utm_campaign,utm_term,utm_content,product_list_limit
Queries parameters excluded from the layered navigation spam filter
Step 5: Configure path filter segment patterns (optional)
Some Magento extensions for layered navigation include filters in layered navigation URLs instead of keeping them as query parameters. If you use one of those extensions, specify that filter in the Path Filter Segment Pattern field to check those URLs against the Magento layered navigation spam filters.
Each filter match is then counted by the Minimum Filter Parameters count and checked against the request limits filters.
Path filter segment patterns for bots blocker if layered navigation extension is used
Step 6: Check if layered navigation spam is blocked
Once you configure the Layered Navigation Protection with Magefan Bot Blocker, check if the filters are working properly without affecting real customers. The configured filters should challenge only excessive requests, while legitimate shoppers should continue using layered navigation normally.
Go to frontend to test layered navigation spam filters:
- Check if category pages load as usual
- Try filter queries on URL patterns you excluded
- Apply a few filters on category pages to test the max request limit per minute.
- Check filters on pagination and search pages
- Try opening a filtered URL in an incognito tab with the filters already applied.
- Check URLs with parameters e.g. utm, that you excluded.
If your activity exceeds any of the limits, you should see the following CAPTCHA verification message.
It means the layered navigation spam filter works properly.
Example of captcha challange based on layered navigation spam filters
Step 7: Monitor spam traffic with the traffic monitor
Once you verify the protection challenges, the Magefan Bot Blocker will monitor and block all layered navigation spam it detects. You can view suspicious traffic via System > Bot Blocker > Traffic Monitor.
Top requested paths targeted by bots and blocked by the bot blocker
And if you want to see more details on what IPs, countries or user agents those requests are coming from, check the IP overview tab at the bottom of the page.
Example of the raw log of requested paths by IPs, countries and user agents
FAQs
How do I stop bots from crawling layered navigation in Magento?
Does noindex stop bots from requesting Magento layered navigation URLs?
How do I prevent bots from generating thousands of Magento filter URLs?
Will Magento layered navigation protection block real customers?
Should I block the IPs generating layered navigation spam?
