Check the answers are good

Search that "seems fine" when you try a few queries can quietly get worse — a model change, a reworded page, a new section that outranks the right one. Evaluations turn "seems fine" into a number you can watch.

What an evaluation is

An evaluation is a set of test questions with a note of which result should come back for each — your own "these are the answers I expect". Search Stack runs every question against your list and scores how well the real results match what you expected. For a docs search, your questions are the ones your users actually ask: "how do I change a field safely?" should land on the Versioning page; "search by picture" on Image search.

You don't have to hand-mark every result. A judge — an LLM you point at the task — can read a question and a returned page and decide whether it's a good answer, so you can score a large set without grading each one yourself.

Why it matters here

A website feed keeps changing under you — that's the point of it. Pages get edited, new ones appear, the crawl re-splits sections. Any of that can shift what ranks first. An evaluation you can re-run turns each of those changes from a silent risk into a check: run it after a change, and if the score drops you know before your users do.

Set one up

In the console, open Insights and create an evaluation against your reference list. Add a handful of real questions and the page each should return, attach a judge if you'd like it scored automatically, and run it. You get a score now, and a baseline to compare against later.

Re-run it whenever you change something — swap the embedding model, restructure your docs, tune the crawl — and watch the number rather than guessing. That habit, more than any single setting, is what keeps a search good as the site behind it grows.

Go deeper: Evaluations, judges, and tracking quality over time in the reference.