Evaluations

An eval set is a saved test of your search quality: named cases pinned to one version of your data, scored by a judge, giving one number — the pass rate.

• A case is one real search — “waterproof running jacket” — with its own standard for a good result.
• A judge is an AI model that reads each result and decides pass or fail against that standard, billed to your own API key.
• Pin a version, then measure every change against it — a new embedding model, a reranker, re-imported records, a setting moved from test to production. If the rate falls, the change hurt quality.

Auto-plays · use Back / Next to step through at your own pace.

A set and its cases

• Criteria belong to the case, not the set. A search case’s judge sees one result at a time, never the search that found it (an Ask case’s judge reads the question with the answer), so one standard cannot cover both “running shoes” (a shoe made for running, in stock, with a size) and “waterproof jackets” (waterproof, not just water-resistant).
• Blank case criteria use the set’s default — its What counts as a pass text. Blank never means “don’t score this case”.

Create an eval set

1. Insights → Evaluations, then New eval set.
2. Pin a target: a list version or a group version. This fixes which records the test searches.
3. Tick Score each result with a judge. Under Grade with, pick the saved credential whose API key pays and optionally that vendor’s model, then write the set’s default What counts as a pass criteria, used by every case that states none of its own — e.g. “the product matches the search and is in stock”.
4. Unticked gives a metrics-only set: no judge calls, just counts of results returned per search, and searches that returned nothing.
5. Optionally turn on auto-run, then click Create.

A new set is created empty and cannot run until it has at least one case. Its Cases dialog opens straight away.

• 25 eval sets on the free plan, 100 on paid, unlimited on Enterprise. Over the limit, create is refused with code 1 (title name).

Add the cases

On the eval set’s row, open the … menu, choose Cases, then click Add case.

• The list shows each case’s name, results judged, search text, and either its own criteria or “Uses the set’s default criteria”. Pencil and bin edit or delete a row.
• 25 cases on the free plan, 50 on paid, unlimited on Enterprise; over it, a new case is refused with code 2 (title name). The same target can be run from several sets.
• Search text up to 1,000 characters; criteria up to 16,000; a set’s default criteria up to 16,000 with at most 16 images; at most 64 text fields and 64 image fields per case.
• Grade an answer instead of results: a case added through the API or MCP with "ask": true runs Ask on its search text and the judge grades the one cited answer. It counts as one result, plus the model call that writes the answer.
  • The answer is written by the set’s judge model on your key. The judge reads the question, the outcome (answered, or refused with the reason) and the cited records.
  • A refusal is graded too, so criteria can say the question should be declined. A question that finds no records is a case with no results.
  • Each Ask case uses one of the account’s 2,000 answers a day; past that, the case fails.

You pick a key, not a judge

• An eval always asks the model the same question — does this result meet the criteria? — so there are no instructions to write.
• Name the credential whose key pays, and optionally one of that vendor’s models; without one you get the vendor’s cheapest tier.
• A judge is separate: a model with reusable, named instructions for filtering live search results. You don’t have to set one up first.
• The judge an eval uses is found and reused per credential and model, so ten sets sharing a key share one judge instead of using up your plan’s judge allowance.

What pinning does, and what it doesn’t

• A pin fixes which records are searched — the same records, not the same results.
• Records, embeddings, model and reranker are read live on every run, which is what lets a re-run catch a drop in quality.
• Two runs a week apart can differ if the model, reranker, judge or records changed in between. Each run records the model, reranker and judge it ran under.
• A group pinned at version 0 (or none) is fixed to its current version number when the set is created.
• The set records when its target and judge were created. If either is deleted and recreated under the same name, the run fails rather than switch to the new one.
• A group pinned to its current version still runs if a list is removed from it without a new version, but the run is flagged membership_drifted.

Run it and read the pass rate

Open Runs & history. The cost is shown up front: how many judge calls this run makes on your key, totalled from your cases; an Ask case counts as two, the answer and its grading. Run now scores every case in the background, one case at a time.

• The pass rate counts results, not cases. A case asking for 10 results contributes 10 verdicts, one asking for 3 contributes 3: five cases returning 25 results, of which the judge approved 6, score 24%.
• A result the judge could not reach a verdict on is an error: left out of the pass rate, not counted as failed. A run with nothing scored has no pass rate.
• A low rate has two causes: the results really are poor, or the cases ask for more results than the data holds — ask for 25 when only 8 records could match and the other 17 drag it down.
• Each run reports whole numbers first — results judged, passed against failed, errors, cases that came back empty — then every case worst first with its own counts, so a bad score points you at the case to look at.
• Runs build a trend line; each point carries its counts and the model, reranker and judge that run used — hover to see them. Only compare points where those match.
• Every case in a run is graded by the version of the judge instructions in force when the run started; an edit made mid-run applies from the next run.
• A run moves Queued → Running → AwaitingJudges → Completed, or ends Failed. AwaitingJudges means every case has been searched and the judge is working through the results.
• A run still Queued or Running after 6 hours, or AwaitingJudges after 34, reads as Failed: The run stopped responding. Start a new run. Reading its status saves it Failed and sends eval.run-completed.
• A Failed run says why, in failure_reason on the run status and next to the run in the console. pass_rate stays null until a run is Completed.
• A run fails when its judge has no pass field (every result would pass), when every judge call errored (The judge failed: <reason>), or when no judge run could start (The judge failed: no judge run could start (…)).
• A run stops if its eval set or account is deleted while it runs; judge calls already under way stop within about 5 seconds.
• A case whose search failed is not a case with no results: the run counts it in failed_queries and leaves it out of the no-results rate. Its row carries search_error and case_name. If every search fails, the run fails.
• When 3 cases in a row find the search target unavailable — an attached index revoked, deleted, unreachable or throttling — the run is retried for about 15 minutes. Only its final attempt stops searching and records the rest with that error. search_error names the cause. Ask cases don’t count toward the 3.
• Each case is one search: it counts toward your plan’s search quota, and on a target with your own embedding model or reranker it makes that call on your key. The estimate counts judge calls only.
• Judge calls start at about two a second: a 50-case set judging 10 results each takes about 4 minutes just to start every call.
• Expect minutes to hours, not seconds. A judge run nothing will finish counts as stopped after 6 hours (34 with a provider batch), so the run can complete sooner than its ~33-hour limit: results the judge never returned count as errors, left out of the pass rate. If no result came back at all, the run fails. The console’s runs dialog updates itself.

A worked example

A products group scores a steady 94%. A new embedding model drops it to 78%; switching back (or fixing it) returns the line to 91% — caught by the next run rather than from a customer complaint.

An illustrative trend: the shape a drop in quality makes when an eval catches it.

Measure every change automatically

• Turn on auto-run and the set re-runs whenever the pinned data changes: after an automatic media re-import, or once a group has finished moving to a new embedding model.
• Each auto-run bills your key; a metrics-only set makes no judge calls.
• An auto-run is skipped while the set already has a run going. Run now and the run route always start one.
• There is no built-in schedule: for a standing check, call the run route from your own scheduler.

Catch a drop

Two webhook events let a release pipeline act on a run without polling.

• The rule: a run has regressed when its pass rate is 0.05 (5 points) or more below the most recent earlier comparable run of the same set.
• A run is comparable when it Completed with a pass rate, every case was processed, and no more than a fifth of searches failed or results errored. Others are never checked and never used as the earlier run.
• The verdict is saved on the run, so run status shows it even if a webhook is lost (table below).
• Both are sent after the run is saved, best effort: they never hold up or fail the run.

Alert in your own monitoring

Finished runs send metrics to your telemetry destinations that have the metrics signal ticked.

• Every eval metric is tagged account, eval_set, model, reranker, judge, trigger and membership_drifted. The two rate histograms also carry status="completed".
• trigger (and a run’s trigger_source) is Manual (console), Api (API or MCP), MediaReimport or ModelConverged.
• The searches a run makes are counted in searchstack.search.* with origin="eval"; the searches a judge makes carry origin="judge".

Sets you made before cases existed

• Sets created before cases held a plain list of searches and one set of criteria. They read those queries back as cases named case-1, case-2 and so on, each with no criteria of its own, so each uses the set’s. Runs score as they always did.
• The first edit, add or delete of a case writes the whole list out as real cases before applying the change. Left alone, it keeps working.

API and MCP

Cases have their own routes, under the set they belong to:

• Create a set with judge = {"credential_name": …, "model": …, "criteria": …} — a credential and an optional model — or judge_name to reuse an existing judge instruction, never both.
• Bootstrap takes source_run_ids of earlier retrieve-then-judge runs: each run’s search becomes a case, and the target and judge come from those runs, so they must share one of each. Runs given explicit candidates are skipped.
  • Identical searches become one case; cases are named case-1, case-2 … and use context_text as their criteria; size is capped at 25; runs with no search text (image-only) are skipped.
• Deleting a set deletes its cases, runs and any judge instructions it created for itself. A run still in progress loses its per-query rows, judge runs and verdicts at once; finished runs’ go shortly after, in the background. Deleting the pinned list or group, or the account, deletes its sets.
• On a case, leave out criteria to use the set’s context_text.
• Editing a case keeps every field you leave out. Send an empty criteria, filter, text_fields or image_fields to clear it.
• A case or set create or edit refused over one field is titled with that field — size, query, context_text.
• Permissions: eval:view reads; eval:create creates sets, bootstraps, and adds or edits cases; eval:run runs; eval:delete deletes sets and cases. See API keys.
• Agents get every operation through MCP:
  • Read: eval_set_list, eval_set_view, eval_case_list, eval_case_view, eval_run_status, eval_run_history.
  • Write: eval_set_create, eval_set_bootstrap, eval_set_delete, eval_case_create, eval_case_edit, eval_case_delete, eval_run.