User-agent: * Disallow: /wp-content/uploads/wc-logs/ Disallow: /wp-content/uploads/woocommerce_transient_files/ Disallow: /wp-content/uploads/woocommerce_uploads/ Disallow: /wp-admin/ # The usual "Allow: /wp-admin/admin-ajax.php" is REMOVED on purpose, and it is the single # biggest crawl-budget win on this site. Server logs for 2026-08-04 (adaptoras.access.log): # # Googlebot requests that day 44,620 # ...to /wp-admin/admin-ajax.php 14,329 (32% of the whole day) # ...of those, HTTP 499 timeouts 792 (805 of all 806 Googlebot 499s) # # Product pages fire ~10 admin-ajax POSTs per render (WP Rocket's LCP beacon, WooCommerce # order attribution, a pricing plugin, the theme's product view). Each one pays the full # ~3s WordPress bootstrap — 63 plugins + 46 mu-plugins — so Googlebot was abandoning about # one in eighteen. # # Safe because nothing indexable is injected by AJAX. Verified on live markup: a category # page serves 127 facet links, 94 hub links and 25 products in the HTML, and a product page # serves its h1, prices and Product JSON-LD server-side. The standard Allow exists for sites # that build content client-side; this one does not. # # AdsBot is unaffected — it obeys only its own group at the end of this file. Disallow: /wp-json/ Disallow: /*?add-to-cart= Disallow: /*&add-to-cart= Disallow: /*?s= Disallow: /*&s= # Google Ads / paid-traffic click parameters Disallow: /*?gad_source= Disallow: /*&gad_source= Disallow: /*?gbraid= Disallow: /*&gbraid= Disallow: /*?wbraid= Disallow: /*&wbraid= Disallow: /*?gclid= Disallow: /*&gclid= Disallow: /*?gad_campaignid= Disallow: /*&gad_campaignid= Disallow: /*?fbclid= Disallow: /*&fbclid= Disallow: /*?msclkid= Disallow: /*&msclkid= # WooCommerce sort and price-range parameters Disallow: /*?orderby= Disallow: /*&orderby= Disallow: /*?min_price= Disallow: /*&min_price= Disallow: /*?max_price= Disallow: /*&max_price= # Filter Everything plugin parameter variants Disallow: /*?yith_wcan= Disallow: /*&yith_wcan= Disallow: /*?filter_ # &filter_ was missing while every other parameter here has both forms. It is reachable: # /sundesh/...?query_type_sundesh=or&filter_sundesh=... appears in the "Crawled - currently # not indexed" export, and ?filter_ alone does not match it. Disallow: /*&filter_ # ?express=1 means "in-stock products only" (inc/perf-typesense-filter-ui.php:439) — a filtered # variant of a page that is already crawlable unfiltered, so it is duplicate by construction. Disallow: /*?express= Disallow: /*&express= # Wishlist / cart / AJAX action parameters Disallow: /*?add_to_wishlist= Disallow: /*&add_to_wishlist= Disallow: /*?remove_from_wishlist= Disallow: /*&remove_from_wishlist= Disallow: /*?wc-ajax= Disallow: /*&wc-ajax= Disallow: /*?customize_changeset_uuid= Disallow: /*?elementor-preview= Disallow: /*?et_fb= # NOTE: /wishlist/, /cart/, /my-account/, /order-tracking/ kept crawlable # so Google can pick up the new noindex meta. Block these in Phase 2. # --------------------------------------------------------------------------- # FilterEverything combinatorial URLs. # # Search Console, Aug 2026: 440k pages "Excluded by noindex" plus 19.5k "Not found (404)" # — about 460k junk URLs against a real catalogue of ~45k sellable products, so roughly ten # crawls of filter permutations for every crawl of something we actually sell. # # These are already noindex or 404, so blocking the crawl strands nothing: they are not in # the index and have no external links. That is the opposite of the out-of-stock products, # where noindex was required precisely BECAUSE the URLs had to be removed from the index — # robots.txt would have frozen them there. Here there is nothing to remove. # # Every rule below was checked against 1,063 indexable device-hub URLs (the FE sitemap), # 2,585 product URLs and all 3,845 URLs that earned a click in the last 28 days: zero matches. # # Two traps these rules are shaped around: # - /cd/oppo/ (slash) is a real attribute archive, 200 + index,follow. /cd-oppo/ (hyphen) # 404s. Only the hyphen form is blocked. # - Products live at /proion/, facets at /proionta/. The "-or-" rule is scoped to # /proionta/ so the product with "-or-" in its slug stays crawlable. # Multi-value selections inside one facet ("...-or-...") — combinatorial, always noindex. Disallow: /proionta/*-or- # Multi-value selections under the OTHER facet roots. Added 2026-08-30. # # The /proionta/*-or- rule above only ever covered one root. Facet URLs also live at # top-level roots (/cd/apple/..., /material/silikoni/...), and those were never # disallowed — note the older rules below use the HYPHEN form (/cd-) which does not # match the SLASH form (/cd/). Access log for 2026-08-30: # # all requests that day 433,546 # ...containing "-or-" 197,206 (45.5% of the entire site) # ...of those, HTTP 200 (full render) 160,283 # top roots: /cd/ 132,229 /material/ 55,932 /kataskeuasths/ 7,260 /seira/ 5,338 # by agent: ClaudeBot 169,108 Googlebot 7,982 Claude-SearchBot 5,517 GoogleOther 5,395 # # robots.txt is the right tool here: ClaudeBot OBEYS it. It made 125,998 requests to # /cd/*-or- and 55,751 to /material/*-or-, but only 25 to the /proionta/*-or- space # that was already disallowed. These roots were simply never listed. # # SEO cost, from analytics_gsc_page (28d, 2026-07-30..08-27, 11,112 site clicks): # every URL containing "-or-" earned 1 click and 330 impressions site-wide (0.009%). # Per root the -or- space earned ZERO clicks and zero impressions. Blocking it returns # crawl budget to real pages, so the expected net effect is positive. # # Scoped per root ON PURPOSE, twice over: # - Single-value facet pages stay crawlable. /kataskeuasths/ alone (brand archives, # no "-or-") earns 241 clicks and 19,092 impressions in the same window. # - NOT written as a global "Disallow: /*-or-": 92 published products have "-or-" # in their own slug (baseus-...-for-smartphone-or-tablet-gray, adidas-or-wavy-...). # A global rule would depend on Allow: /proion/ winning by longest-match in every # crawler's parser. It is not worth betting 92 product pages on that. # # To add a root later: find it with # grep -oP ' GET /\K[^/ ]+(?=/[^ ]*-or-)' /var/log/nginx/adaptoras.access.log | sort | uniq -c | sort -rn Disallow: /arithmos-thiron/*-or- Disallow: /availability/*-or- Disallow: /battery-capacity/*-or- Disallow: /brand/*-or- Disallow: /case-addons/*-or- Disallow: /case-cat/*-or- Disallow: /case-category/*-or- Disallow: /case-color/*-or- Disallow: /cd/*-or- Disallow: /charger-type/*-or- Disallow: /eidos-smart-device/*-or- Disallow: /eisodos/*-or- Disallow: /ekdosi-bluetooth/*-or- Disallow: /ekdosi-html/*-or- Disallow: /ekdosi-usb/*-or- Disallow: /gia-siskevi/*-or- Disallow: /idaniko-gia/*-or- Disallow: /idos-mpatarias/*-or- Disallow: /idos-paixnidiou/*-or- Disallow: /idos-siskevis-katharismou/*-or- Disallow: /kartes-sim/*-or- Disallow: /katallilo-ilikia/*-or- Disallow: /kataskeuasths/*-or- Disallow: /katigoria-ethernet/*-or- Disallow: /katigoria-isxios/*-or- Disallow: /katigoria-proiontos/*-or- Disallow: /katigoria-prostasias-othonis/*-or- Disallow: /material/*-or- Disallow: /megethos-power-bank/*-or- Disallow: /megisti-analisi/*-or- Disallow: /megisti-isxis/*-or- Disallow: /mhkos/*-or- Disallow: /mikos-kalodiou/*-or- Disallow: /mikos-kalodiou-usb/*-or- Disallow: /modelo/*-or- Disallow: /montelo/*-or- Disallow: /periexi-kalodio/*-or- Disallow: /prostheta-xaraktiristika/*-or- Disallow: /protokollo-fortisis/*-or- Disallow: /protokollo-grigoris-fortisis/*-or- Disallow: /protokolo-epikoinwnias/*-or- Disallow: /ram/*-or- Disallow: /seira/*-or- Disallow: /simvati-priza/*-or- Disallow: /simvato-me-drone/*-or- Disallow: /simvato-me-niimbot/*-or- Disallow: /sindesi-akrodekton/*-or- Disallow: /siskevi/*-or- Disallow: /supported-devices/*-or- Disallow: /systhma-prosarmogea/*-or- Disallow: /texnologia/*-or- Disallow: /thires/*-or- Disallow: /thires_usba/*-or- Disallow: /tipos-akoustikon/*-or- Disallow: /tipos-kalodiou/*-or- Disallow: /tipos-prosarmogea/*-or- Disallow: /tipos_aisthitira/*-or- Disallow: /tipos_fortisti/*-or- Disallow: /tipos_usb/*-or- Disallow: /usb-c-thires/*-or- Disallow: /varos-power-bank/*-or- Disallow: /visma-sindesis/*-or- Disallow: /vysma-sindesis/*-or- Disallow: /xarakthristika-akoustikon/*-or- Disallow: /xarakthristika-prosarmogea/*-or- Disallow: /xrwma/*-or- Disallow: /xwrhtikotita/*-or- # Root-level facet shells with no category context. Every one of these returns 404. Disallow: /cd- Disallow: /seira- Disallow: /modelo- Disallow: /brand- Disallow: /case-color- Disallow: /case-cat- Disallow: /case-addons- Disallow: /availability- Disallow: /montelo- Disallow: /siskevi- # Cosmetic facets stacked onto a hub — noindex, and combinatorial with each other. Disallow: /proionta/*/case-color- Disallow: /proionta/*/case-cat- Disallow: /proionta/*/case-addons- Disallow: /proionta/*/availability- # /device-finder/ exists only as a Google Ads landing surface. It is a React app, so there is # nothing for an organic crawler to read, and it earned 10 clicks in the 28 days to 2026-08-01. # Its sitemap submission is removed below for the same reason. # # This Disallow is safe for the ads themselves: AdsBot-Google ignores "User-agent: *" by design # and obeys only a group named for it, so the explicit AdsBot groups at the end keep landing-page # crawling (quality score, policy checks) working while Googlebot stays out. Disallow: /device-finder # Giant "-and-" device lists under the colour archives, and the legacy render parameter. Disallow: /xrwma/*/modelo- Disallow: /*?metro_legacy_render= Disallow: /*&metro_legacy_render= # Attribute archives that are already `noindex, follow` and have earned nothing in Search # Console. They can never rank, so every fetch is crawl budget spent on a dead end. Measured # over the 28 days to 2026-08-01 (terms / GSC impressions / GSC clicks): # # /gia-siskevi/ 14,339 terms 0 impressions 0 clicks <-- the big one # /kathgoria/ 299 terms 0 0 (404s anyway) # /mobile-brands/ 125 terms 5 1 # /kwdikos-siskeuhs/ 122 terms 0 0 (404s anyway) # /sundesh/ 46 terms 0 0 # /xoritikotita-mpatarias/ 43 terms 0 0 # # Deliberately NOT blocked: # /kataskeuasths/ — index,follow and earns 19,490 impressions / 249 clicks. Leave it alone. # /seira/ — noindex, but still shows 141 impressions / 7 clicks. Not worth the risk. # # These pages are `follow`, so blocking removes one internal path to products. That is # acceptable here because all 45,232 sellable products are in the XML sitemap (46,086 URLs # submitted across 83 product sitemaps) and a URL-inspection sample found 93% already indexed. Disallow: /gia-siskevi/ Disallow: /kathgoria/ Disallow: /mobile-brands/ Disallow: /kwdikos-siskeuhs/ Disallow: /sundesh/ Disallow: /xoritikotita-mpatarias/ Sitemap: https://www.adaptoras.gr/sitemap_index.xml Sitemap: https://www.adaptoras.gr/sitemaps/filtereverything-sitemap.xml # adaptoras.cy is the same WordPress install under a second hostname (mu-plugin 36), and nginx # serves this ONE static file as the document root for both. Everything above therefore applies to # .cy too, which is exactly what we want for the Disallow rules — the facet-space and attribute # archives are the same pages there. Only the Sitemap lines needed a .cy counterpart, and they # cannot be made per-host without turning robots.txt into PHP (there is no robots_txt filter here; # all 69 Disallow rules live only in this file), so .cy's index is simply listed alongside. A # crawler reading either host gets its own correct sitemap; the other host's is cross-domain and # ignored unless separately verified. Yoast generates the .cy index correctly — verified 2026-08-10, # 99 entries all on the .cy host — and metro-child/inc/sitemap-cache.php keys its cache on # HTTP_HOST, so the two hostnames cannot serve each other's cached sitemap. # # Deliberately NOT advertising the filtereverything sitemap for .cy: that one is built as static # XML with .gr URLs baked in, so pointing crawlers at it from .cy would hand them wrong-host URLs. # The file is reachable at /sitemaps/ on either host, but nothing advertises it on .cy. Sitemap: https://www.adaptoras.cy/sitemap_index.xml # Ads landing pages must stay fetchable. AdsBot follows only its own group, so this re-opens # everything for it — including /device-finder/, which the * group above blocks. User-agent: AdsBot-Google Allow: / User-agent: AdsBot-Google-Mobile Allow: /