Faceted navigation (the filters for size, colour, brand, price and material on category pages) is essential for shoppers and one of the most common crawling problems on ecommerce sites. Each filter combination can become its own URL, so a modest catalog can expose hundreds of thousands of near-identical pages to search engines. The fix is to decide which combinations deserve to be found in search, give those clean indexable URLs, and keep every other combination out of crawling, not just out of the index.
How filters multiply URLs
The arithmetic is what surprises people. A single category with five colours, eight sizes, four price bands and three sort orders already has several hundred single-choice combinations. Once shoppers can select more than one value per filter, and parameters can appear in any order (?size=9&colour=black and ?colour=black&size=9), the number of distinct URLs grows much faster. Multiply that by every category and you get a URL space far larger than your actual catalog.
Search engines discover these URLs through the filter links on your pages. Google’s guidance on faceted navigation describes the result: crawlers spend time on URLs that add nothing, and new or updated pages that matter can be discovered and refreshed more slowly.
Does crawl budget apply to your store?
For small sites, usually not much. Google’s crawl budget guide is written for large sites and for medium sites whose content changes quickly. Many stores fall into those groups once filter URLs are counted, even with a few thousand products, because the crawler sees the URL space, not the product count.
Signs that it applies to you:
- Search Console’s Crawl Stats report shows a large share of requests going to parameterised URLs.
- The page indexing report lists many filter URLs as “Crawled – currently not indexed” or as duplicates.
- New products take a long time to appear in search results.
- Server logs show crawlers requesting deep filter combinations that no shopper would use.
Decide which combinations deserve to rank
Some filter combinations are real searches: “black leather boots”, “oak dining tables”. Those should not be filters at all from a search point of view; they should be proper subcategories or landing pages with clean URLs, their own title and heading, and useful copy. We describe the test we use (search demand, enough products, a genuinely different choice) in ecommerce category page SEO.
Everything else, including all sort orders, view options, price ranges and multi-select combinations, should work perfectly for shoppers and be invisible to crawlers.
The options, and what each one actually does
| Approach | Stops crawling? | Stops indexing? | Notes |
|---|---|---|---|
| Filters that do not create crawlable links (for example, applied with URL fragments or form controls rather than plain links) | Yes | Yes | The most effective option; filters still work for shoppers |
| robots.txt disallow for filter parameters | Yes | Mostly, but blocked URLs can still be indexed without content if linked | Simple and effective; do not block the indexable subcategory URLs |
| Canonical tag to the unfiltered category | No | Often, as a hint | URLs are still crawled; best as a supporting signal |
noindex on filter pages |
No | Yes | Pages must be crawled to see it; does not save crawling |
nofollow on filter links |
Partly | No | Treated as a hint; other links can still lead crawlers in |
| Return a 404 for combinations with no products | Reduces waste | Yes | Stops empty result pages being crawled repeatedly |
Google’s own guidance favours preventing crawling of filter URLs you do not want in search, through robots.txt or by not exposing them as crawlable links, and returning a 404 for empty filter combinations. robots.txt controls crawling, not indexing, which is why it pairs well with clean subcategory URLs that are allowed.
A setup that works for most stores
- Indexable subcategories for the combinations with search demand, with clean paths (
/boots/black-leather) linked from the category and navigation. - Filters that change the product grid without creating crawlable links, or that use parameters you disallow in robots.txt.
- A consistent parameter order and lowercase values, so the same filter state never has two URLs.
- Sort and view parameters disallowed and canonicalised to the unsorted category.
- 404 for empty combinations, rather than an empty grid with a 200 status.
- The XML sitemap listing categories, indexable subcategories and products only.
Platform notes
- Shopify: storefront filtering uses URL parameters on collection pages, and tag-based filtering creates additional collection URLs. Themes can control which filter links are crawlable, and robots.txt can be customised through the theme’s
robots.txt.liquidtemplate. - BigCommerce: faceted search generates parameterised category URLs; review which facets are exposed as links and use robots.txt rules for the rest.
- WooCommerce: layered navigation, attribute archives and filter plugins can each create indexable URLs. Check what your theme, filter plugin and SEO plugin output together, since they often disagree about canonicals.
A worked audit: sizing the problem
When we audit faceted navigation on a store, we answer four questions before recommending anything:
- How big is the filter URL space? Crawl the site with filter links followed, with a sensible limit, and count URLs by pattern: clean categories, single filters, multiple filters, sorts, pagination. The ratio of filter URLs to real categories and products shows the scale.
- What are crawlers actually requesting? Server logs, filtered to verified search engine crawlers, show where crawling goes. Search Console’s Crawl Stats report gives a sampled view if logs are not available.
- What is indexed? The page indexing report and a site search for parameter patterns show whether filter URLs are appearing in results, and which ones.
- What earns traffic? Search Console queries and landing pages reveal filter URLs that already rank. Those are the candidates for proper subcategories, and the URLs you must not block until they have a new home.
The answers usually point to a small set of subcategories to create and a few URL patterns to block. On stores where filter URLs dominate crawling, we then watch the Crawl Stats report over the following weeks to confirm crawling shifts toward products and categories.
What about filters built with JavaScript?
Filters that update the grid with JavaScript can still create crawlable URLs if they also rewrite the address bar and render plain links for each option. Check the HTML the server sends: if every filter option is an <a href> with a unique parameter URL, crawlers can follow them whatever the page does after it loads. Equally, make sure the default product grid for each category is in the server-rendered HTML, so crawlers see the products without running any filter code.
Before you change anything
Changing crawl rules on a live store needs care: blocking the wrong pattern can hide real categories. Our checklist:
- List the URLs that currently receive organic traffic, including any filter URLs that rank. Those need a proper indexable home before you block the pattern they use.
- Test robots.txt rules against a sample of real URLs, both ones to block and ones to keep.
- Change one rule at a time and watch the Crawl Stats and page indexing reports for a few weeks.
- Keep internal links pointing at the clean category and subcategory URLs.
Filter changes often come up during a replatform, when URL structures change anyway. If that is your situation, plan the filter URLs as part of the redirect map so ranking filter pages are not lost in the move.
Our ecommerce SEO team audits crawl logs and Search Console data to size the problem, designs the subcategory and filter rules, and implements them with your theme. Where the filter code itself slows the category pages down, our speed optimization work covers that too.