Judges

A judge is a quality check for your search results: an AI model that reads each result and decides whether to keep or drop it, following instructions you write, with every call billed to your own API key.

• The model is one you already use — Anthropic’s Claude, OpenAI’s GPT, Google’s Gemini, or a model deployed to your own Microsoft Foundry resource.
• Judges run after a search, in the background, and the verdict comes back in a fixed format: keep it, or drop it.
• A judge removes poor matches; it does not reorder results the way a reranker does.
• Use one for a standard relevance scores cannot check: “is this profile really a match?”, “is this product family-friendly?”, “keep only products with a price and a real description”.

Auto-plays · use Back / Next to step through at your own pace.

The two halves: a judge and its instructions

• A judge alone decides nothing. The instructions are what runs — what you try out, what a run executes, and what evals and watches name.
• One judge carries several sets of instructions: relevance, safety and is-it-in-stock on one model and one key, so changing the key is one edit, not three.

Create a judge

1. Models → Judges → Add Judge, then a provider: Anthropic (Claude), OpenAI (GPT), Google Gemini or Microsoft Foundry. Calls run on your own key — your models, your limits, your bill.
2. Name it, or leave it blank to be named after the provider and model, e.g. anthropic-claude-haiku-4-5. Name it (fast, careful) for two judges on one model.
3. Pick the model that will do the reading.
4. Pick a credential (a saved API key). Keys saved for that vendor come first; Enter a new key… pastes a fresh one, saved for reuse. Where custom endpoints are enabled, Advanced points a new key at a proxy or compatible service instead of the provider’s address — see Credentials.
5. Microsoft Foundry only: the Resource your model is deployed to. Foundry has no default address, so it is always asked for; the field offers resources saved for your Foundry models and rerankers, and a reused Microsoft credential brings its resource, skipping the field.
6. Add. The key is checked with a real call before saving, and re-checked regularly; the health dot beside the judge shows a problem first.

The judge appears as a card, with no instructions yet.

Add instructions

On the judge’s card, click Add instructions:

1. Name it — relevance, safety, whatever the check is. Evals, watches and API calls use this name.
2. Write the instructions in plain English: “Pass only if the product is in stock, has a price, and the description says what it is.” The result’s text is added automatically.
3. Define the output schema: a JSON Schema for the answer’s shape, such as {"pass": true, "reason": "..."}. The judge must reply in exactly that shape, so verdicts are data you can act on, not free text. Suggest drafts one from your instructions (billed to your key), often naming the pass field; it is editable, and nothing saves until Add.
4. Optionally a pass field: a true/false field such as pass. A result is then kept only when it comes back true; blank keeps every result for you to read the verdicts yourself.

Add as many sets as you need; each appears under its judge, with its pass field beneath the name.

Tune it with Try

Open a set of instructions’ ⋮ menu and choose Try before relying on it.

• Give it a context (what the searcher wants, e.g. “a hiking partner near Leeds”) and one candidate, plus image URIs (web addresses of pictures) if the judge should look at pictures.
• The answer is immediate: a green Pass, grey Fail or amber Errored badge, and the full JSON verdict.
• Nothing is saved — keep adjusting the instructions and schema until the verdicts look right.

History: every edit is a version

• Instructions are versioned: each edit saves a new version and keeps the previous one read-only, so the wording a run was judged against is never lost.
• ⋮ → History lists every version newest-first; expand one to read its instructions, schema and pass field.
• Restore copies that version’s content forward as a new current version — it never rewinds the history.

Editing and deleting

• Edit judge (the card’s ⋮ menu) changes the saved credential the judge uses and its custom endpoint. Provider and model are fixed: to change model, delete the judge and add a new one.
• To swap in a new key, update it once on the Credentials tab and every judge using it follows.
• Edit on a set of instructions changes the instructions, schema or pass field, and adds a new version.
• Deleting a judge deletes the instructions on it; deleting a set of instructions leaves the judge intact.

Where judges get used

• Watches: a watch can name a set of instructions and add its own “Only alert me if…” criteria (e.g. “only if it is under £100 and in my size”), so only new records that meet the standard become alerts.
• Evals: in Insights → Evaluations, an eval set picks which AI model judges it and carries its own pass criteria, re-running a fixed set of searches to turn quality into a single pass-rate number.

API and MCP

AI agents get the read-only half through MCP: judge_list, judge_view, judge_instruction_list, judge_instruction_view, judge_instruction_version_list, judge_instruction_version_view and judge_run_status.