Evaluations
An eval set is a saved relevance test for your search. You pick a handful of real searches, pin them to one version of your data, and let a judge score the results. Run it and you get a single number — the pass rate — that says how good your search is right now. Run it again after any change and compare. If the number falls, the change hurt quality.
That is the habit evals give you: pin a version, then measure every change against it. Swap an embedding model, attach a reranker, re-import a batch of records, promote a config from test to production — re-run the set and the pass rate tells you, in seconds, whether search got better or worse. It is search quality as a number you can watch, not a feeling you argue about.
Auto-plays · use Back / Next to step through at your own pace.
Create an eval set
- Go to Insights → Evaluations and click New eval set.
- Pin a target — a list version or a group version. This fixes which records are in scope for the test.
- Pick a judge and, optionally, the criteria it scores each result against. No judge? You still get cheap hit and zero-result counts — a “metrics only” set.
- List the searches your users actually make, each with a size (how many top results to score). Every result scored is one judge call on your key — the dialog totals the cost up front.
- Optionally arm auto-run so the set re-runs itself when your data changes.
- Click Create.
What pinning does — and what it doesn’t
Pinning fixes which records are searched — not the results. Your records, embeddings, model and reranker are all read live every time the set runs. That is deliberate: it is exactly what lets a re-run catch a regression. If pinning also froze the model, changing the model couldn’t change the score — and catching that change is the whole reason the eval exists.
So a pin gives you the same records to search, not the same results. Two runs a week apart can score differently if you changed the model, the reranker, the judge, or the records in between — and each run records the model, reranker and judge it actually ran under, so the history stays readable.
Run it and read the pass rate
Open Runs & history. Before anything runs, it shows the cost up front — how many judge calls this run will make on your key. Hit Run now and it scores every search in the background, a few at a time. When it finishes you get two numbers:
| Metric | What it tells you |
|---|---|
| Pass rate | Of every result the judge looked at, how many passed. Your headline quality number. |
| Zero-result rate | How many of your searches returned nothing at all — a blunt gap you want low. |
Run the set a second time and the two build a trend line you can watch over the weeks.
A worked example
Say your products group scores a steady 94% week after week. One morning you switch it to a new embedding model and re-run the set — the pass rate drops to 78%. That is a regression, and you caught it in seconds, not from a customer complaint. You roll the model back (or fix it), re-run, and the line returns to 91%. Without the eval, that dip is invisible until someone notices bad results in production.
An illustrative trend — the shape a regression makes when an eval catches it.
Measure every change automatically
You don’t have to remember to re-run. Arm auto-run and the set re-runs itself whenever the pinned data changes — after an automatic media re-import, or after a group’s embedding-model change finishes converging. Each auto-run scores on your judge’s key (or is free for a metrics-only set), and a set never stacks a second run while one is already in flight.
Judges do the scoring
The pass/fail verdict on each result comes from a judge — an AI model you configure with your own key and your own rubric. The same judge that filters your live search results scores your eval sets. See Judges for how to set one up and tune it.