Check the answers are good

Evaluations turn “seems fine” into a number you can watch.

What an evaluation is

An evaluation is a set of test questions with a note of which result should come back for each. Search Stack runs every question against your list and scores how well the real results match. For a docs search, use the questions your users actually ask: “how do I change a field safely?” should land on Versioning; “search by picture” on Image search.

A judge — an LLM you point at the task — reads a question and a returned page and decides whether it's a good answer, so you can score a large set without grading each one by hand.

Why it matters here

A website feed keeps changing under you: pages get edited, new ones appear, the crawl re-splits sections. Any of that can shift what ranks first. Re-run the evaluation after a change and a drop in the score tells you before your users do.

Set one up

1. Open Insights, switch to the Evaluations tab, and choose New eval set.
2. Target your reference list, and pick the saved API key that pays for the scoring if you want it graded automatically.
3. The set is created empty. Open its … menu, choose Cases, and add a handful of real questions — each a case with its own bar for a good answer.
4. Run it. You get a score now, and a baseline to compare against later.

Re-run it whenever you change something — swap the embedding model, restructure your docs, tune the crawl.

Go deeper: Evaluations, judges, and tracking quality over time in the reference.