User-agent: * Allow: / # Layered navigation exposes an unbounded URL space: every filter permutation is # a distinct URL, which misses the cache and costs a full Magento render. The # filtered pages already carry noindex,nofollow, so no crawler needs them. # Blocking every query string covers new filter attributes automatically; # pagination stays crawlable. Crawlers that ignore this are rejected with 429 at # the edge (see edge01 default.vcl). Keep the sitemap reference relative: this # document root serves several domains. Disallow: /*? Allow: /*?p= Sitemap: /sitemap.xml