Evaluations

An eval set is a saved test of your search quality: named cases pinned to one version of your data, scored by a judge, giving one number — the pass rate.

• A case is one real search — “waterproof running jacket” — with its own standard for a good result.
• A judge is an AI model that reads each result and decides pass or fail against that standard, billed to your own API key.
• Pin a version, then measure every change against it — a new embedding model, a reranker, re-imported records, a setting moved from test to production. If the rate falls, the change hurt quality.

Auto-plays · use Back / Next to step through at your own pace.

A set and its cases

• Criteria belong to the case, not the set. The judge sees one result at a time, never the search that found it, so one standard cannot cover both “running shoes” (a shoe made for running, in stock, with a size) and “waterproof jackets” (waterproof, not just water-resistant).
• Blank case criteria use the set’s default — its What counts as a pass text. Blank never means “don’t score this case”.

Create an eval set

1. Insights → Evaluations, then New eval set.
2. Pin a target: a list version or a group version. This fixes which records the test searches.
3. Tick Score each result with a judge. Under Grade with, pick the saved credential whose API key pays and optionally that vendor’s model, then write the set’s default What counts as a pass criteria, used by every case that states none of its own — e.g. “the product matches the search and is in stock”.
4. Unticked gives a metrics-only set: cheap counts of results returned per search, and searches that returned nothing.
5. Optionally turn on auto-run, then click Create.

A new set is created empty and cannot run until it has at least one case. Add them from the set’s own row.

Add the cases

On the eval set’s row, open the … menu, choose Cases, then click Add case.

• The list shows each case’s name, results judged, search text, and either its own criteria or “Uses the set’s default criteria”. Pencil and bin edit or delete a row.
• 25 cases on the free plan, 50 on paid, unlimited on Enterprise. The same target can be run from several sets.

You pick a key, not a judge

• An eval always asks the model the same question — does this result meet the criteria? — so there are no instructions to write.
• Name the credential whose key pays, and optionally one of that vendor’s models; without one you get the vendor’s cheapest tier.
• A judge is separate: a model with reusable, named instructions for filtering live search results. You don’t have to set one up first.
• The judge an eval uses is found and reused per credential and model, so ten sets sharing a key share one judge instead of using up your plan’s judge allowance.

What pinning does, and what it doesn’t

• A pin fixes which records are searched — the same records, not the same results.
• Records, embeddings, model and reranker are read live on every run, which is what lets a re-run catch a drop in quality.
• Two runs a week apart can differ if the model, reranker, judge or records changed in between. Each run records the model, reranker and judge it ran under.

Run it and read the pass rate

Open Runs & history. The cost is shown up front: how many judge calls this run makes on your key, totalled from your cases. Run now scores every case in the background, a few at a time.

• The pass rate counts results, not cases. A case asking for 10 results contributes 10 verdicts, one asking for 3 contributes 3: five cases returning 25 results, of which the judge approved 6, score 24%.
• A result the judge could not reach a verdict on counts as not passed.
• A low rate has two causes: the results really are poor, or the cases ask for more results than the data holds — ask for 25 when only 8 records could match and the other 17 drag it down.
• Each run reports whole numbers first — results judged, passed against failed, errors, cases that came back empty — then every case worst first with its own counts, so a bad score points you at the case to look at.
• Runs build a trend line; each point carries its counts and the model, reranker and judge that run used — hover to see them. Only compare points where those match.

A worked example

A products group scores a steady 94%. A new embedding model drops it to 78%; switching back (or fixing it) returns the line to 91% — caught in seconds rather than from a customer complaint.

An illustrative trend: the shape a drop in quality makes when an eval catches it.

Measure every change automatically

• Turn on auto-run and the set re-runs whenever the pinned data changes: after an automatic media re-import, or once a group has finished moving to a new embedding model.
• Each auto-run bills your key; a metrics-only set is free.
• A set never starts a second run while one is still going.

Sets you made before cases existed

• Sets created before cases held a plain list of searches and one set of criteria. They read those queries back as cases named case-1, case-2 and so on, each with no criteria of its own, so each uses the set’s. Runs score as they always did.
• The first edit, add or delete of a case writes the whole list out as real cases before applying the change. Left alone, it keeps working.

API and MCP

Cases have their own routes, under the set they belong to:

• Create a set with judge = {"credential_name": …, "model": …, "criteria": …} — a credential and an optional model, not a judge name.
• On a case, leave out criteria to use the set’s context_text.
• Agents get the same operations through MCP: eval_case_list, eval_case_view, eval_case_create, eval_case_edit, eval_case_delete, eval_set_list, eval_set_view, eval_run, eval_run_status, eval_run_history.