Judges

A judge is a quality check for your search results. You write the instructions; an AI model you already use — Anthropic’s Claude or OpenAI’s GPT, with your own API key — reads each result and returns a structured verdict: keep it, or drop it.

Judges run after a search, in the background — a precision filter, not the live reordering a reranker does. Use one to hold results to a bar that relevance scores can’t check on their own: “is this profile really a match for what the searcher asked for?”, “is this product family-friendly?”, “does this document actually answer the question?”

Auto-plays · use Back / Next to step through at your own pace.

Create a judge

1. Go to Models → Judges, click Add Judge and pick a provider — Anthropic (Claude) or OpenAI (GPT). Judge calls run on your own key: your models, your limits, your bill.
2. Write the instructions — what the judge should decide about each result, in plain English. The result’s text is added to the prompt automatically.
3. Define the output schema — a JSON Schema describing the answer you want back. The judge is forced to reply in exactly that shape, so verdicts are data you can act on, not prose.
4. Optionally name a pass field — a true/false property in your schema. When set, results are kept only when it comes back true. Leave it blank to keep every result and read the verdicts yourself.
5. Click Add. Your API key is verified with a live call before the judge is saved, and the connection is re-checked regularly afterwards — the health dot next to the model shows any problem first.

Tune it with Try

Before wiring a judge into anything, open its ⋮ menu and choose Try. Give it a context (what the searcher wants) and one candidate (a single result), and the judge answers live — a green Pass or grey Fail badge plus the JSON verdict. Nothing is saved, so you can iterate on your instructions until the verdicts feel right.

A different candidate — wrong location, none of the interests — comes back Fail, and in everyday use a result like that is the one a judge drops. Only results that clear your bar reach your users.

A worked example

Say you run a dating app, and someone searches for a hiking partner. Your search already returns the profiles that mention hiking or the outdoors — but “mentions hiking” is not the same as “a genuine match”. A judge reads each returned profile against the searcher’s brief and keeps only the ones that really fit.

Here the search is “experienced hiking partner near Leeds who loves the outdoors”, and the judge is told to be strict: a good match must share the location and name at least one shared interest. Six profiles come back, and the judge gives each a verdict:

Three of the six clear the bar. Those are the ones your users actually see — same city, a real shared interest:

The other three are quietly dropped: the right city but no shared interest, a keen hiker too far away, and someone who is neither.

Judges power evals

The same judge that scores results here can also power evals — a way to watch your search quality over time. In Insights → Evaluations, an eval set replays a fixed set of queries and uses a judge to score every result, turning quality into a single pass-rate number you can keep an eye on as you change models or data. See the Evaluations guide for the full walkthrough.