Loading...
searchstack.dev
Sign In

Connect your website

The finished example searches Search Stack's own reference pages. Nobody exported or uploaded them — a website feed crawled them from one address and keeps them in step. That's this step.

What a website feed is

A feed connects an outside source and mirrors it into a file Search Stack keeps up to date — you never upload anything by hand. A website feed's source is your own site: you give it one web address, it reads the pages under that address, and it refreshes on a schedule as your site changes. There's nothing to install on your site, and nothing for you to run.

The address rule

You describe which part of your site to keep searchable with a single address and one rule: everything that starts with what you type is included. The * goes on the end, right after a / — "and everything below here".

You enterWhat's included
https://example.com/docs/*Every page under /docs/, however deeply nested.
https://example.com/*Your whole site.
https://example.com/pricingJust that one page (no * means exactly that address).

Connect it

Go to Feeds (under Sources in the sidebar) and choose Connect feed, then Website.

  1. Give the feed a name — for the demo it's reference-site.
  2. Enter the site address. For our reference docs that's https://searchstack.dev/reference/*.
  3. Click Find pages. The feed does a quick read of your site and shows the pages it found, all ticked. Untick anything you don't want (a changelog, an internal note), Add a page it missed, then click Connect.

The review list works in rules, not a fixed list: pages you publish under your address later are picked up on their own, so the feed keeps itself current without you maintaining an inventory. It reads politely too — it respects your robots.txt, identifies itself as SearchStackBot, and paces its requests.

Or from code

The same two moves over the API. First, preview the pages under your address (nothing is saved yet):

POST https://api.searchstack.dev/feed/Demo/web/discover
X-API-Key: {your key}
Content-Type: application/json

{ "url_pattern": "https://searchstack.dev/reference/*" }
{
  "pages": [
    { "url": "https://searchstack.dev/reference/versioning", "title": "Versioning" },
    { "url": "https://searchstack.dev/reference/fields",     "title": "Fields" }
  ],
  "truncated": false
}

Happy with the list? Connect the feed. url_excludes and url_seeds are optional — they're the "untick" and "add a page" from the dialog:

POST https://api.searchstack.dev/feed/Demo/connect
X-API-Key: {your key}
Content-Type: application/json

{ "feed_name": "reference-site", "provider": "web", "url_pattern": "https://searchstack.dev/reference/*" }
{
  "feed_name": "reference-site",
  "provider": "web",
  "store_name": "default",
  "file_path": "feeds/reference-site.ndjson",
  "sync_interval_minutes": 30
}

Go deeper: The address rule, how pages are found, and what ends up in each record in the reference.

Top