Telemetry destinations

Send the numbers Searchability already records about your search into your own monitoring tools — Datadog, Grafana, Honeycomb, New Relic or anything that accepts OpenTelemetry (OTLP).

• A continuous stream of metrics, traces and logs; a webhook is one event at a time.
• Auth headers are the only secret. Encrypted when stored, never shown again. Custom destinations can have none.
• Found under Insights → Telemetry.

Add a destination

1. Insights → Telemetry → Add Destination. Pick a provider (Grafana Cloud, Honeycomb, New Relic), which fills in the header name and, except Grafana Cloud, the endpoint. Or pick Custom for any other OTLP endpoint.
2. Give it a Name: letters, numbers and single hyphens, stored lower-case. It cannot be changed later.
3. Set the endpoint. HTTP (port 4318): the base URL — /v1/<signal> is added for you. gRPC (port 4317): host and port, no path. The transport is under Advanced and cannot be changed later.
4. Paste your key into the auth header (required for a provider; optional for Custom). Add more headers if your backend needs them.
5. Signals (under Advanced) default to traces, metrics and logs; untick any you don’t want, but keep at least one. Click Create.

Providers

Endpoint and header rules

• https only. No internal, loopback or link-local addresses.
• Transport (protocol): otlp_grpc or otlp_http, in any case. Any other value is refused with a message naming both.
• No credentials, query string or fragment in the URL — put secrets in a header.
• Header names must be valid HTTP header names, used once each. Values cannot be empty or contain control characters (line breaks included); surrounding spaces are trimmed.
• Values must be plain ASCII: no accented letters or curly quotes.
• gRPC: header names may only contain letters, digits, ., _ and -.
• Refused names: Host, TE, Connection, Transfer-Encoding, and any starting content-, grpc- or :, or ending -bin.
• Editing a destination: re-enter each header (name and value); only a provider’s header name is filled in. Leaving every value blank removes all headers.

The save-time check

Create and Edit send one empty export and refuse the save if it fails.

• HTTP: sent to the first selected signal's path — /v1/traces, else /v1/metrics, else /v1/logs.
• gRPC: calls Export on the first selected signal's service, in the same order. Unimplemented fails only when that service is missing.
• gRPC answered with a non-200 HTTP status (a proxy or web server): judged by the HTTP rules below, except 429 passes and 400, 404, 413 or 415 fail as not a gRPC receiver.

• The health dot is that same check, run every 6 hours and again whenever a search service, cache, media store or judge on the account is saved, or Resume syncing is chosen on a search service or media store.
• The dot shows the endpoint answers, not that data is arriving.
• One failed check turns the dot amber. Three failed checks in a row, counted at most once an hour, turn it red (usually 12–18 h) and raise an Account health issue.
• A failing or degraded destination shows the reason under its name.
• A successful create or edit marks the destination healthy at once.
• Saved headers that can no longer be decrypted turn the dot red: “The saved auth headers can't be read. Edit the destination and enter them again.” The destination can still be viewed, edited and deleted.

What is sent

• Traces and logs of a request on another account's list (as a contributor or shared-list reader) go to no one's destination.

Metric families

Listed in full under Exported metrics. Names start searchstack.; every metric carries account.

Search outcome is success, or the reason the search failed:

• search.* counts every search made through SearchStack: your API and console searches, and the searches your evals, judges, watches and Ask run. origin tells them apart: api (your own traffic, the default), eval, watch, judge, ask, probe.
• On an attached Azure AI Search index, searches your app sends straight to Azure are not counted. Use Azure’s own metrics for those.
• origin="probe" is our own readiness check. It appears only on the account it is pointed at.
• provider is the search backend key (for example search_stack_v1, azure_attached_v1). It is set only once the search service is resolved.
• account, list and group read (unresolved) until the name is found. account is lower-case.
• scope is list or group. vector_search, reranker and cache are what the request asked for (defaults false, true, true), not what ran. cache_hit (true/false) appears only when the result cache was used.
• search.zero_results counts successful list and group searches that returned nothing. Zero-result rate of your own traffic = search.zero_results{origin="api"} / search.requests{origin="api",outcome="success"}; without origin="api" it also counts eval, watch, judge and Ask searches.
• import.running and import.stalled are refreshed every 10 minutes. No series, or 0, means none. An account reports its own 0 for one sweep when its imports finish or a stall clears; a reading older than 12 minutes is dropped. In Grafana take max by (account).
• OpenAI embedding calls are counted once, with account.
• A rule pattern that runs out of time adds a gate.failures point with reason="pattern_timeout".
• Ratio histograms (pass rate, zero-result rate, watch precision) use 0–1 buckets. Read them with histogram_quantile.
• Series names follow the standard OTLP→Prometheus naming rules.

The search-quality story

The quality metrics — search.zero_results, judge.verdicts, eval.* and watch.precision — carry the eval set, watch or verdict each number came from; search.zero_results carries origin.

• Pass rate fell — alert when the median of searchstack.eval.run.pass_rate drops below its recent normal. The eval.run-regressed webhook and the searchstack.eval.regressions counter use one fixed rule — see Catch a drop.
• Zero-result rate spiked — alert when the zero-result rate above rises past your limit (say 2%).
• A watch got noisy — alert when searchstack.watch.precision falls.

Check it works

1. Run a search on any list.
2. In your backend, look for searchstack.search.requests{account="<your account>"}. Prometheus-style backends (Grafana) name it searchstack_search_requests_total. Allow 2–3 minutes after creating the destination.
3. Metrics unticked (for example Datadog)? Look for a trace with a search.engine span instead.

Limits

• Best effort: each batch gets one attempt with a 10 s timeout. A batch your backend refuses is dropped.
• A new or changed destination takes up to ~90 s to start receiving.
• A deleted destination keeps receiving for up to ~90 s.
• Under heavy load, batches can be dropped.
• Destinations are managed in the console only. The API and MCP account_get list them read-only, as telemetry_sinks in both SDKs' AccountResponse.
• Outbound calls to your endpoints (feeds, webhooks, BYO providers, destination checks) carry trace context but no baggage header.

Start from a starter dashboard

Exported metrics → Download starter dashboard builds one from these metrics. These are starters — check every panel in your backend.

• Grafana, New Relic — import the JSON directly in their UIs.
• Datadog — the board charts metrics only. It fills only when they reach Datadog through your own OpenTelemetry Collector or the Datadog Agent (both convert cumulative to delta). With metrics unticked it stays empty.
• Honeycomb — the file is a Boards API request body you POST to api.honeycomb.io/1/boards with your own Configuration key.
• The Grafana file also has a Search quality row: eval pass rate and zero-result rate per eval set, live zero-result rate (your API traffic only), judge verdict split, eval runs by status, regressions in the window, and watch precision per watch.