Telemetry destinations
A telemetry destination streams the operational data Searchability already records about your search workload — how fast searches run, how many results they return, how often they come back empty, how your embedding model, reranker and judges are performing — straight into your own monitoring stack. It is the same idea as a webhook, but for a continuous stream of metrics, traces and logs rather than one event at a time. You point it at Datadog, Grafana, Honeycomb, New Relic or any backend that speaks OpenTelemetry (OTLP), and your search behaviour shows up next to the rest of your systems.
Two things make this worth setting up. First, you see your search health where your on-call already looks — no new dashboard to remember. Second, we never hold your monitoring credentials: you give a destination one auth header, it is encrypted at rest, and that is the only secret involved.
Auto-plays · use Back / Next to step through at your own pace.
Add a destination
- Go to Insights → Telemetry and click Add Destination. Pick your provider — Grafana Cloud, Honeycomb, New Relic — or Custom for any other OTLP endpoint. A provider pre-fills the endpoint and the right header name for you.
- Set the endpoint (the receiver’s base URL — the
/v1/<signal>path is appended for you) and choose the OTLP transport: HTTP/protobuf (port 4318) or gRPC (port 4317). - Paste your key into the auth header. It is encrypted at rest and never shown again.
- Tick the signals this destination should receive — traces, metrics, logs — and click Create. A health dot on the row then shows whether exports are getting through.
The three signals
| Signal | What you get |
|---|---|
| Traces | A span per search request — the timeline of a query through embedding, retrieval, rerank and judging. Use it to see where a slow search spent its time. |
| Metrics | The counters and histograms below — rates and latencies you chart and alert on. This is the signal the search-quality story is built on. |
| Logs | Structured log records for notable events, correlated with the traces above. |
Send only what you need — a destination that just powers alerting might take metrics alone; one for debugging might add traces.
What the metrics tell you
With the metrics signal on, Searchability exports a fixed catalog of instruments — visible any time under Exported metrics. Every metric carries an account attribute, so on a multi-tenant setup you group or filter by tenant. Names follow the searchstack.* convention:
| Metric | What it is |
|---|---|
searchstack.search.duration | How long search requests take (histogram). |
searchstack.search.requests | How many searches are handled. |
searchstack.search.zero_results | Searches that returned nothing — a rise means real queries are coming back empty. |
searchstack.embedding.duration / .failures | How your embedding model performs, and when a call fails. |
searchstack.rerank.duration / .failures | The same for your reranker. |
searchstack.eval.run.pass_rate | Pass rate of a completed eval run — tagged with eval_set, model, reranker, judge. |
searchstack.judge.verdicts | Judge pass/fail verdicts, tagged by provider and model. |
A pass rate is a per-run observation pushed at completion, so it is a histogram — read it with histogram_quantile (a median over the window), and read a rate as zero_results / requests. Series names follow the standard OTLP→Prometheus normalisation.
The search-quality story
Most of these metrics any monitoring tool could roughly approximate from the outside. The eval and judge metrics are the ones that cannot come from anywhere else: they carry the pin and verdict tags from your eval sets, so a dashboard can show a pass-rate line per eval set, a live zero-result trend, and the judge verdict split — a real relevance SLO, not just an uptime chart. Two alerts mirror the CI gate:
- Pass rate fell — page when the median of
searchstack.eval.run.pass_ratedrops more than a few points below its recent baseline. This is the same drop theeval.run-regressedwebhook fires on, now visible as a graph your on-call watches. - Zero-result rate spiked — page when
zero_results / requestsrises above your tolerance (say 2%).
The webhook gate catches a bad change at deploy time; a quality SLO catches it any time — including a slow drift nothing deployed to cause. Both read the same signal.
Start from the built-in dashboard
You don’t have to build the first board by hand. Exported metrics → Download starter dashboard generates one from this same catalog for Grafana, New Relic, Datadog or Honeycomb. Grafana, New Relic and Datadog import the JSON directly in their UIs; the Honeycomb file is a Boards API request body you POST with your own key. Its Search quality row gives you the pass-rate line, zero-result trend, judge split and a regression stat out of the box — then adapt it.
A worked example
Your team runs Grafana. You add one destination pointed at your Grafana Cloud OTLP endpoint, paste in the API key, and tick metrics. You open Exported metrics, download the Grafana starter dashboard, and import it. Within minutes your existing Grafana shows a pass-rate line per eval set beside your app’s other panels, and you wire an alert to page on-call when the median pass rate falls five points. Now a relevance regression — whether a bad deploy caused it or a slow drift did — reaches the same on-call, through the same tool, as any other production problem.