Millions of people go through millions of pages they find on Google daily. If you want your Magento website to be within that number and get discovered by potential customers, you need to get into the Google index. Otherwise, your chances of getting organic traffic range from zero to none, which is basically the same thing.
To get Google to index your website you need to know how the index works and how to submit a page to Google. So, today we'll cover this along with the most common reasons why Google doesn't index your website and how to fix that.
Don't get overwhelmed just yet. It is actually not that hard to grasp the concepts of the Google indexing process. We promise.
Post Contents [hide]
Warning: we'll start from the basics. So if you've already generated a sitemap for your Magento store, just move to the other sections.
How Google Finds Your Content?
Before you learn how to get Google to index your website, you need to know how the index works in the first place. The process comes down to 4 steps, which we've simplified for better understanding.
- Discover — first Google needs to learn your website exists. It does that through the sitemaps you submit to it and the backlinks your website gets from other websites.
- Crawl — crawling is a process when a Google bot, or spider, visits your website and downloads pages to get them to the list of known pages.
- Process — at this stage, key information about your pages is extracted and prepared for indexing.
- Index — indexing is when the data extracted from your pages during processing is moved to a search index database. It is essentially a library of web pages from which Google pulls the search results.
How to Index Your Website with Google?
Getting Google to index your website is pretty easy. You just need to make sure Google can find your pages. As we've stated before, if you've already submitted a sitemap to Google, it will discover your pages eventually. However, are you willing to wait?
Since most of you want to see new or updated pages in search nearly immediately, we'll show you a way.
But first things first.
Request a crawl
Go to your Google Search Console (GSC) property and paste the requested URL in the top search panel. The console will retrieve data about the URL.
If your URL is not in Google Index, hit the Request Indexing button so Google will schedule a crawl for your page. That's the easiest way to index your Magento website with Google.
Note: in some cases, Google returns the error message saying that your page can't be crawled. Usually, it's because of the noindex meta robots. We'll talk about it later.
Submit XML sitemap to Google
XML sitemap is a map with all the important pages of your website. Google uses it to discover your pages and index them.
Since Magento allows you to generate XML sitemap automatically, all you need to do is submit it to Google. By default your Magento website sitemap, should be available via the sitemap.xml path. e.g. https://yourwebsite.com/sitemap.xml.
So, just take your website sitemap URL and go to the Sitemap section in the GSC. Paste the sitemap path there and click Submit.
Once you submit the sitemap, Google will start discovering more of your pages. But note that it's a long process, especially if you have thousands of pages.
Use Google Indexing API
Although having an XML sitemap is more than enough to get indexed by Google, it is a long process. If you want to get indexed by Google faster, you need the Google Indexing API.
This way you get more control of what gets indexed and when. Besides, you can send multiple pages for indexing in bulk, without relying solely on a sitemap.
It might sound too technical. But you can still configure it for your store using the . It enables you to send indexing requests directly from the admin and covers the technical part of the process.
So to send indexing requests through the Indexing API with this extension, you just need to go to any page and click the corresponding button.
To grasp how easy it is to set up the indexing API, check out the guide on Google Indexing API configuration.
How Long Does It Take to Get Indexed in Google?
When you try to index your website with Google, the main question is how long will it take. My advice is — just be ready to wait and don't expect Google to index your website too fast.
Note: remember that Google bots need to crawl and process your pages before they move them into the index. And it takes a while, considering the amount of pages you have.
If you have a new website with under 500 pages it will get indexed in about two or three weeks. For larger websites — 500 or a couple of thousand pages — the indexing period extends to a few months.
However, if you just submitted a new page, it should get indexed in about a week.
Nonetheless, when you use Google Indexing API, you don't have to wait for Google to discover your pages through a sitemap. You prompt Google to crawl your pages (and index them correspondingly) much faster.
Why Google Doesn't Index Pages?
If you've done all the steps to index your website on Google, and you still don't see your page in search results, there are a couple of reasons for that.
So now let's analyse why Google doesn't index your Magento pages.
Pages missing from the sitemap
Since Google heavily relies on the XML sitemap to discover your pages, it's the first thing to check, if some pages aren't indexed. If you exclude a page from the XML sitemap Google can still discover it through other ways.
So it is unlikely to be the only reason pages are not indexed. There should be something else.
Pages blocked in robots.txt
You use the robots.txt file in Magento to tell Google what pages it can and can't crawl. Correspondingly, if a page is blocked in the robots.txt it won't be indexed.
You can use the GSC, the Pages section, to discover pages excluded from indexing due to the crawl block.
Meta robots tags
If Google has crawled your website already, but some pages are still not indexed it might be because of robots meta tags. The noindex tag, in particular, tells Google that, though it has discovered the page, it is not allowed to index it.
Here you can also use the GSC to discover all pages with noindex tags under Pages seciton.
Canonicals
Canonical tags tell Google about the main version of the page when you have pages with similar content or the same page available via different URLs.
You use canonicals to tell Google what pages you want indexed. Thus, if you use the rogue canonical tag, you tell Google to index a version that doesn't exist or that you don't want to be indexed.
In GSC this issue is marked as "Alternate page with proper canonical tag" and can also be found under Pages.
These are the most common reasons why Google might not index your pages even though you want it to. Troubleshoot these in GSC to fix the issues and get your pages into Google Index faster.
How to Exclude Pages from Google Indexing?
Now you know how to index your Magento website with Google and the reasons why Google might not do so. However, there is one more thing — what if you decide to remove a page from search results?
In this case, you need to follow a different approach closely related to the tools we've just described. Get more details in our guide on how to remove pages from Google indexing to get the results with no technical background.
