Cleaning

Cleaning tidies your records as they are imported into a list — whether they arrive by upload, from a media store, or through a feed. It never changes your source; everything it produces is materialised only in the SearchStack index, so search sees a cleaner catalogue than the raw data.

Cleaning has two layers. The first is deterministic and always on. The second is LLM extraction, which reads each record’s copy to pull out facets, and runs only when the list has a cleaning credential.

Deterministic cleaning (always on)

Every import is tidied with a set of rules that need no AI and cost nothing:

• De-duplication. Records with the same identity are collapsed into one. Records that look like the same product but aren’t an exact match are flagged for you to confirm rather than merged silently.
• Value normalisation. Facet and category values are rewritten to a canonical form — “UNITED KINGDOM” becomes “United Kingdom”, “VIET NAM” becomes “Vietnam”.
• Tag-noise stripping. Internal bookkeeping values are removed from your public facets, so shoppers never see them.
• Category mapping. Free-text categories are mapped onto the Google Product Taxonomy using a built-in dictionary.

LLM extraction (optional)

When a list has a cleaning credential, cleaning also reads each record’s copy (its title, description, tags and so on) and extracts structured facets you can filter on — for example vegan, gluten-free, and a Google category for the long tail the deterministic dictionary didn’t map. Extracted facets only ever fill gaps: a value you already supplied is never overwritten, and every extracted value is tagged with its provenance so you can audit it.

You never write a prompt. The extraction task is generated for you; all the credential provides is the API key and endpoint the extraction runs on. Without a credential, extraction is simply off for that list and only the deterministic layer runs.

Assign a credential to a list

1. Save a credential first: add a BYO judge, reranker or embedding model with an API key, and it becomes available to pick here.
2. Open a list’s ⋮ menu and choose Cleaning credential (or set it when you create the list), then pick a saved credential. Choose None to turn extraction off.

Cost: batch vs per-record

Credentials marked Recommended — Anthropic and OpenAI on their default endpoint — extract through the provider’s batch API, at roughly half the price. Other credentials still work; they extract with a per-record call instead.

Extraction never blocks an import. It runs in the background, and results are cached by content, so an unchanged record is only ever extracted once — re-importing it costs nothing. A record whose copy changes is re-extracted; everything else is served from the cache.