Website Feed

A website feed keeps the pages of your own site — documentation, a help centre, a knowledge base — searchable, automatically. You give it one web address; it reads the pages under that address and turns them into searchable records, refreshing on a schedule as your site changes. No export, no upload, no code — and nothing to install on your site.

This page covers what’s specific to the website feed: the address you give it, how pages are found, and what ends up in each record. For how feeds work in general — syncing, binding a list, deleting — see the Feeds reference.

Connect a website

Go to Feeds (under Sources in the sidebar), click Connect feed and choose Website.
Give the feed a name, choose which media store the synced file should live in (your built-in store by default), and enter the site address — see the address rule just below.
Click Find pages. The feed does a quick read of your site and shows you the list of pages it found, all ticked. Untick anything you don’t want, add any it missed, then click Connect.

From here the feed keeps feeds/<name>.ndjson up to date in your media store; point a list at that file with keep-in-sync switched on and your pages are searchable — the Feeds page walks through that step.

The site address

You describe which part of your site to keep searchable with a single address, and one simple rule: everything that starts with what you type is included.

You enter	What’s included
`https://example.com/docs/*`	Every page under `/docs/` — at any depth, however deeply nested.
`https://example.com/*`	Your whole site.
`https://example.com/pricing`	Just that one page (no `*` means exactly that address).

The * goes on the end, right after a / — it means “and everything below here”. That’s the whole rule: there are no other wildcards and nothing to learn. The address must be a full one starting with https://, on a normal public domain.

Choosing which pages to include

When you click Find pages, the feed reads your site the way a visitor would — starting from your address and following the links between your pages — and shows you what it found. This list is where you fine-tune, and it works in rules, not a fixed list: pages you add under your address later are picked up on their own, so the feed keeps itself current without you maintaining an inventory.

Untick a page you don’t want indexed — a changelog, say, or an internal note — and it’s left out.
Add a page the feed didn’t find. Some pages aren’t linked from anywhere the crawl can reach; add such a page and the feed reads it and follows its links, so an otherwise-unlinked section comes along with its starting page.

A couple of things worth knowing about how pages are found:

The feed indexes pages a visitor can reach by following links — the same pages a search engine would find. A page linked from nowhere won’t be found on its own; that’s what Add a page is for.
It reads politely: it respects your site’s robots.txt, identifies itself as SearchStackBot, and paces its requests. If your robots.txt blocks a page, the feed leaves it alone.
The review shown at connect time reads a first batch of pages so it stays quick; if your site is large it tells you it’ll keep discovering the rest after you connect. The full read happens on the first sync.

What ends up in each record

Each section of a page becomes one searchable record — split on the page’s headings — rather than one giant record per page. That means a search lands the reader on the part of the page that answers them, not just the page. The feed describes its own fields, so these are created for you — nothing to map:

Field	What it holds
`name`	The page’s title, plus the section’s heading — e.g. “Getting started — Installation”.
`content`	The section’s text, with the site’s navigation, header and footer stripped out — used for matching and ranking.
`url`	A link straight to that section of the page, so a result opens exactly where the answer is.
`page`	The page the section came from — handy as a filter to group a page’s sections together.

A note on freshness: syncing is periodic (as often as every 5 minutes, 30 by default), so a just-published page becomes searchable on the next sync rather than the instant it goes live.

A worked example

Say your documentation lives at https://docs.example.com/ and you want it searchable — and answerable by an assistant. You connect a website feed with the address https://docs.example.com/*, review the page list, and connect. The feed mirrors your pages into feeds/docs.ndjson in your built-in media store. You create a docs list, import from that file, and turn on keep-in-sync. From now on, when you publish or edit a page, the feed refreshes the file and your docs search updates itself — new pages appear, edits update in place, removed pages drop out — with no export, no upload, and no code.