Query optimisation

A query rule reads a search before it runs. It spots a format in what someone typed — a postcode, an order number, a barcode — and turns that part of the query into a constraint rather than leaving it to be matched as words.

Query rules are the mirror image of gates. A gate decides what a record looks like on the way in; a query rule decides what a search means on the way out. They sit next to each other on the list header for that reason: that icon opens this page, the shield opens gates.

The problem they solve

With no rules, a search matches the text exactly as it was typed. Somebody searching flat 2 zz1 1zz is searching for four words, and a record for that address is competing with every record that happens to contain “flat” or “2”. The postcode — the most precise thing in the query, and the only part the customer was certain about — counts for no more than the rest.

A UK postcode rule reads that query as “flat 2”, narrowed to postcode ZZ1 1ZZ. Same words typed, a different question asked.

The same applies to anything with a shape: an ISBN, a VIN, an IBAN, an email address, a number plate. Each one identifies something exactly, and each one is a handful of meaningless tokens to a text search.

Rules belong to a list

Unlike a gate, a rule is created on the list rather than on your account. That is because a rule binds a format to a field, and the field only means what it means on this list — a postcode facet on your properties list is not the postcode on your customers list. Open a list and choose the query-rules icon in its header.

A rule can only target a facet. Narrowing is something the search engine does with a facet, so offering a searchable-only field here would let you build a rule whose constraint could never be applied. If a list has no facets yet, add one first — the dialog will say so.

What a match does

Every rule reaches one of three conclusions when it recognises a format:

Narrows — the results are filtered to that value. Right for a format that repeats across many records: a postcode, a date, a price.
Pins — that one record goes to the top, with the normal results beneath it. Right for a format that identifies exactly one thing: an ISBN, an IBAN, a VIN, an order number.
Ranks up — the match is scored higher and nothing is excluded.

The difference between narrows and pins is the choice worth getting right, and it is decided by how many records share a value rather than by the format itself. Pinning is the safer of the two: if the pinned record is not there, the ordinary search still ran underneath and the customer loses one row. A narrow that matches nothing returns a confident empty page.

You do not have to work this out. Each shipped format arrives with the correct answer already chosen — a postcode narrows, an IBAN pins — and the dialog shows the choice as a sentence about what the customer will experience, so you can change it if your data disagrees.

How much evidence a rule needs

Recognising a format is not the same as knowing the value exists. ZZ1 9ZZ is a perfectly well-formed postcode that nobody lives at, and narrowing to it returns nothing at all. One setting — three positions — says how far a rule should go:

Cautious — only acts on a value your data actually contains. Never wrong; costs you the occasional match while the values are still being read.
Balanced — acts unless your data says the value does not exist. The default, and right for almost every rule.
Bold — acts on the shape alone, even when the field is known not to hold the value.

Bold paired with narrows is the only combination that can return a confident empty result set, and the console says so on the control rather than leaving you to find out. Paired with pins it is harmless, which is why the identifier formats ship set to it: a miss costs the pinned row, not the page.

This setting matters most where the shape is weak. 2019 is a Sydney postcode and also a year, and no pattern can separate them — but in a list of Australian addresses it is overwhelmingly a postcode, and you are the only one who knows that. Change the setting from the rule’s ⋮ menu, which offers the two you are not on — Cautious, Balanced, Bold — each with what it means underneath. Turning it up only ever adds constraints, so it is safe to try.

The order rules run in

Queries meet rules top to bottom, and the first rule to claim part of a query wins. The rules below it see only what is left. So the position column is real, and the order is the answer to two rules wanting the same words — a rule reading four digits as a postcode and another reading them as a year cannot both have them.

Move a rule with Move up / Move down in its ⋮ menu. The rows animate to their new positions, so a reorder reads as a movement rather than a jump.

On, test only, off

Every rule is in one of three states:

On — it changes live searches.
Test only — it runs alongside the real search, changes nothing, and records what it would have done.
Off — it does nothing.

Test only is the point of the whole page. A rule that changes how every search of your list is read is not something to switch on hopefully: put it in test only, leave it a week, then read what it did. Every search behaves exactly as it does today while you wait. The badge is amber rather than green for the same reason — a green badge on a rule doing nothing to live searches would be the one place being wrong about that matters most.

You can start a new rule in test only from the Try it without changing anything? question in the add dialog.

What a rule has been doing

Choose What has this been doing? from any rule’s ⋮ menu to see the last seven days:

How often it fired — or would have fired, for a rule in test only — as a share of all searches.
How many of the values it matched your data does not actually hold. This is the number that decides it: those are the searches that would have returned nothing.
The median number of results the searches it fires on currently return.

If too few searches have hit the rule to be worth reading, it says so before showing any figure — a percentage from four searches looks exactly as convincing as one from four thousand. A rule in test only can be switched on, or discarded, straight from this panel.

What you will not find is a “340 results → 4 results” comparison. Running the narrowed search to find out would double the cost of every search the rule touches, on a path budgeted in milliseconds.

Work out my rules

Work out my rules reads the records already in your list, works out what each facet actually holds, and adds the rules your own data argues for. A field of UK postcodes gets a postcode rule; a field of thirteen-digit barcodes gets a barcode rule.

It also reads fields whose values match no format we ship. Where a field repeats enough to be a set to choose from — four countries across a thousand records, a dozen brands, a handful of categories — it gets a rule built from its own values, so searching “scotland” narrows to the country rather than hunting for the word.

Fields of plain numbers are deliberately left alone. A rule on a building number or a quantity would read any number in a search as that field, so “4 bedroom” becomes building number 4 — and no setting prevents it, because 4 really is one of your values. The report says so when it skips one, and you can still add the rule yourself if your customers do search that way.

It runs on its own after an import, so this may already have happened by the time you first open the page — and if a run is going while you are looking, the page says so and the rules appear as they are created, with Add a rule paused until it finishes. Pressing the button asks for it now, and is never turned away by the pacing that keeps a ten-thousand-record import to a single run.

The report names the fields it skipped and why, which is the half worth reading: a postcode column that got no rule needs to say whether we failed to recognise it or decided against it, because those call for opposite responses from you.

Anything added this way is labelled with the evidence behind it — you will always be able to tell a rule you did not write from one you did, which is what makes turning it off a real option.

Try a query

The box below the table answers the question the page exists for: how will this query be read? Type a query, choose Read it, and you get back what actually reaches the search engine and what became a constraint. Nothing is searched and nothing is saved.

It is the fastest answer to “why did my search return that?”. It shows rules in test only as well, so a rule can be checked against a real query before it is ever switched on. If a rule changed since the answer on screen was worked out, it says so rather than showing you an interpretation that no longer happens.

The formats we ship

Eighteen formats come with your account, each one already set to narrow or to pin as its cardinality demands.

These narrow — many records share a value:

UK postcode, US ZIP code, Canadian postal code and Australian postcode — written with or without the space, and a UK postcode is recognised across two words as readily as one.
Date (written YYYY-MM-DD), IP address, and Price (an amount like £19.99, exact — ranges such as “under £20” are not a rule).

These pin — each value identifies one record:

Email address, Phone number (international), UK phone number.
ISBN-13, ISBN-10, Barcode (EAN-13) — with or without hyphens.
IBAN, Identifier (UUID), MAC address.
Vehicle identification number and UK vehicle registration.

The Australian postcode is the one to think about: four bare digits are also a year and also a quantity. If your records hold both, set that rule to act only on values it can confirm.

A rule on your own values

Choose One of this field’s own values and the rule matches search terms against the values already in the field — your brands, your categories, your cities. There is no format to get wrong, so it only ever acts on a value you really have, and it is fixed to the cautious setting for that reason: the field’s values are the whole test.

This is the rule to reach for when someone searching dyson cordless should have “dyson” read as the brand rather than as a word that happens to appear in a description.

A format of your own

Your own part numbers, internal codes, ticket references — formats nobody ships. Choose Something else — I’ll describe it, say what the values look like in your own words, and a model drafts the pattern for you.

You are then shown which of your own values it matched and which it missed, and nothing is created until you approve it. This costs a model call, so it uses one of your saved credentials.

There is no box for writing a pattern by hand, on purpose: a console that asked for one would put every format we do not ship out of reach of exactly the people whose formats we do not know. If you would rather supply a pattern directly, the API accepts one — and so do the client packages, which cover the whole of this page:

await client.QueryRules.create("acme", "parts", {
  name: "part-number",
  target_field: "part_number",
  pattern: "^ACME-\\d{4}-[A-Z]$",
  state: "Shadow",            // watch it for a week before it counts
});

// Nothing is searched and nothing is stored.
const read = await client.QueryRules.test("acme", "parts", { query: "acme-4471-b blue" });

Every call on this page is on client.QueryRules in both the TypeScript and .NET packages — listing, creating, reordering, switching on and off, and reading what a rule has been doing.

Add a rule

Open the list and choose the query-rules icon in its header.
Choose Add a rule.
Answer What does the format look like? — a shipped format, this field’s own values, or one you describe.
Answer Which field holds it? — any facet on the list.
Everything else is filled in for you. Change What should a match do? if your data disagrees, and choose test only if you would rather watch it before it counts.

The Name comes from your two answers. Change it if the list needs a second rule over the same field — a billing postcode and a delivery postcode are two rules, not one.

Notes

Rules are read on every search of the list, including searches through a group the list belongs to.
Nothing about a rule is retroactive and nothing is rewritten: a rule changes how queries are read, never what your records say.
Deleting a rule takes effect on the next search. Searches already answered are not revisited.
A rule that fires on a value your data does not hold is the failure mode to watch, and What has this been doing? is where it shows up. Reach for Cautious before reaching for Off.