Loading...
searchstack.dev
Sign In

API Feed

The API feed is the do-it-yourself feed: give us one HTTPS URL that returns your records as JSON, and we keep a searchable list in sync with it — no connector to wait for, no export, no upload. If your system of record has a read API (or you can expose one), this makes it searchable today.

It is deliberately the simplest feed there is. We fetch your endpoint on a schedule and stage the response body exactly as it comes back; the same importer that reads a file you upload yourself reads it. We never re-shape or re-map your data — so what you get out is precisely what your endpoint returns. This page covers what’s specific to the API feed: the shape your endpoint must return, optional authentication, and how deletions work. For how feeds work in general — syncing, binding a list, deleting — see the Feeds reference.

Connect an endpoint
  1. Go to Feeds (under Sources in the sidebar), click Connect feed and choose API endpoint.
  2. Give the feed a name and paste your endpoint URL. If it needs authentication, fill in the optional auth header (name and value); leave both blank for a public endpoint. Choose which media store the synced file should live in (your built-in store by default).
  3. Click Connect. We call your endpoint there and then, so a wrong URL, a rejected credential, or a response that isn’t JSON is caught immediately rather than failing quietly later — and the first sync starts right away.

From here the feed keeps feeds/<name>.ndjson up to date in your media store; point a list at that file with keep-in-sync switched on and your records are searchable — the Feeds page walks through that step.

What your endpoint must return

A single GET of your URL must return your records as one of:

  • a JSON array of record objects — [{ … }, { … }];
  • a single JSON object;
  • NDJSON — one JSON object per line.

That’s the only requirement. Each object’s properties become fields on a record, and their roles are inferred just as they are for a JSON or NDJSON file you upload yourself — there is nothing to map. If your endpoint returns HTML, plain text, or an error page, the connect step tells you so rather than importing rubbish.

Full fidelity: the common format

For exact control — your own record IDs, declared field roles, and deletion handling — have your endpoint return the common format: NDJSON whose first line is a manifest. It’s the same format the other feeds produce and the same one you can upload by hand, so nothing here is API-feed-specific:

{"searchstack_format":1,"source":"crm","snapshot":true,"fields":{"summary":"vectorised","status":"facet"}}
{"id":"cust-1","name":"Ada Lovelace","summary":"Long-standing account","status":"active"}
{"id":"cust-2","name":"Grace Hopper","summary":"New this quarter","status":"active"}
  • id gives each record a stable identity, so renaming a record updates it in place instead of creating a duplicate.
  • fields declares each field’s role (searchable, facet, resource, or vectorised) so provisioning is exact rather than guessed.
  • snapshot: true turns on deletion reconciliation — see below.

If you need a shape we don’t produce for you — joining two systems, renaming fields, filtering rows — that shaping is your endpoint’s job, not a mapping screen in Search Stack. Return the common format and you get full fidelity; return a plain array and it still works, just with guessed field roles and no deletions.

Keeping search in step with deletions

By default an API feed is additive: each sync adds new records and updates changed ones, but never removes anything (exactly like re-uploading a file). If your endpoint returns the complete set of records every time, add "snapshot": true to its manifest and the feed will also deactivate records that are no longer in the response — so a record you delete at the source drops out of search on the next sync. Only use snapshot when the response really is the whole set; a partial or paged response under snapshot would deactivate everything it left out.

Authentication

Authentication is optional and deliberately minimal: a single header sent on every request. Fill in the header name (for example Authorization or X-API-Key) and its value (for example Bearer … or your API key). The value is stored encrypted and never shown again — to change it, remove the feed and reconnect. The endpoint must be https so the header is never sent in the clear.

For security, the URL must be a public, named host on the standard https port — IP addresses, localhost, intranet names and custom ports are refused. If your data lives somewhere private, put a small public endpoint in front of it that returns the records.

How change is detected

On each poll we fetch the endpoint and compare the response to the last one; a byte-for-byte identical response stages nothing, so an unchanged source costs a fetch and nothing more. Any edit or deletion changes the response and triggers a fresh sync. Syncing is periodic (as often as every 5 minutes, 30 by default), so an edit becomes searchable within minutes rather than instantly — and large sources are best sharded across several narrower endpoints, one feed each.

A worked example

Say your CRM has a reporting API. You add a small read-only endpoint — https://api.acme.com/searchstack/customers — that returns your customers as common-format NDJSON with snapshot: true, protected by an Authorization header. You connect it as an API feed and paste the header. The feed mirrors the response into feeds/customers.ndjson in your built-in media store. You create a customers list, import from that file with keep-in-sync on, and now your CRM is searchable — add a customer and it appears, delete one and it drops out — with no export, no upload, and no connector to wait for.

Top