Quillet
Menu
Menu

Search index

6 min read

The search index is what the delivery API's search route reads: it holds the text of every published, searchable document, follows each publish by itself, and is rebuilt by an owner when it has fallen out of step.

Why the CMS keeps the index for your site

A site's search box needs an index of the content. If the site keeps that index itself, it has to walk the whole environment at every build. Quillet keeps one for you instead. A site asks /d/v1/<project>/<env>/search a question, and the answer says which published pages match, under which heading, with a snippet of the text around the words.

The index is built from published content only, so search can never show a reader a draft. It is a copy and never the source of truth, because every entry can be rebuilt from the documents. That is why its one maintenance task is a rebuild, and why the switch that turns search off deletes the index instead of hiding it. A content manager decides what is findable with the Searchable setting on a document or a folder. An owner decides whether the project has search at all.

What goes into the index and when it changes

What is indexed

  • Only published versions. A save changes nothing in the index. A publish, an unpublish or a delete does, and the index changes before the publish request answers.
  • Only searchable documents. Searchable on the document decides, and Inherit takes the folder's setting. A document's own setting takes effect when the document is saved. A folder's setting takes effect when the folder's settings are published, because what readers can find is part of what they are served.
  • A Markdown document gives one entry for the whole page and one for each ## and ### section, so a hit can link to the heading through its anchor. Code blocks and inline code are included, while image descriptions and raw HTML are not.
  • A JSON document gives one entry with every string in it, skipping references and file pointers. A plain-text document gives one entry with its text.
  • Folder settings (_index) and components (_schemas/…) are never indexed.
  • Each locale is indexed in its own language where MongoDB can stem it: Danish, Dutch, English, Finnish, French, German, Hungarian, Italian, Norwegian, Portuguese, Romanian, Russian, Spanish, Swedish and Turkish. Every other language is indexed as whole words.
  • Each environment has its own index. A delivery key searches the environment it belongs to.

Long pages are cut, not refused. A section's text is indexed up to 8 KB and the whole page's up to 32 KB. Sections past the 50th have no entry of their own, though their text is still in the page's entry.

The project switch

Search is on for every project unless an owner turns it off. Turning it off deletes the project's whole index, in every environment, before the request answers. From then on the search route answers 404, so a site does not show "No results" for every question. Turning it back on rebuilds every environment in the background, and hits come back over the following seconds.

The switch is the search key in the project's settings, {"search":{"enabled":true}}. Writing settings replaces all of them, so send back whatever else the project's settings hold, or that is lost.

Rebuilding

A rebuild walks every published document of one environment, writes its entries again, and removes every entry it did not write. Its answer says how many documents it considered and how many entries the index now holds. Running it twice writes the same thing.

The index updates itself after every commit. It does not heal on its own if the CMS stops between a publish and the index write. A page can then be missing from search, or a withdrawn page can still be found, until someone rebuilds. A withdrawn page found this way is a hit that leads to a 404, not a leak: its body is never in a search answer, only a snippet of at most 180 characters. A rebuild is also the answer after you change the project's default locale, since that can change what a folder's setting resolves to.

Rebuilding is an owner's work, done with the dashboard or a personal token. It is limited to five times an hour per environment, however many owners press the button.

How to

Dashboard

To turn search on or off:

  1. Open Settings › Project. Only an owner sees the settings there.
  2. Under Search, tick or clear Let sites search this project's published content. It saves as soon as you change it.
  3. Clearing it asks Turn search off?: "Turning search off deletes this project's search index. Turning it back on rebuilds it." Press Turn search off.

To rebuild the index:

  1. On the same row, press Rebuild search index. The line under it names the environment it rebuilds, such as "Rebuilds Prod.", because the dashboard always rebuilds the default environment.
  2. When it is done, it reports "Indexed 42 documents." A sixth rebuild in an hour shows "You can rebuild the index five times an hour. Try again later."

To keep one document out of search:

  1. Open the document. In the Publishing card, set Searchable to Off.
  2. Save. The document leaves the index at once, with no publish needed.

API

Ask a question the way a site does, with a delivery key for the environment:

curl -s -u '<keyId>:<secret>' \
  'http://localhost:3000/d/v1/<project>/<env>/search?q=<words>'

Each hit has the document's key, its title, the heading and anchor it was found under (null for a page-level hit), a snippet and a score. total counts the matches in the window searched.

Rebuild one environment's index with an owner's personal token. Unlike the dashboard's button, this route reaches any environment:

curl -s -u '<tokenId>:<secret>' -X POST \
  'http://localhost:3000/api/v1/projects/<project>/envs/<env>/search/reindex'

It answers {"documents":…,"entries":…}. To turn search off for the project, write its settings; this example is for a project whose settings hold nothing else:

curl -s -u '<tokenId>:<secret>' -X PATCH \
  'http://localhost:3000/api/v1/projects/<project>' \
  -H 'Content-Type: application/json' \
  -d '{"settings":{"search":{"enabled":false}}}'

The first command now answers 404 Not found. Send "enabled":true to turn search back on and start the rebuild.

Limits

  • There is no fuzzy matching and no prefix matching. publi does not find "publishing", and a misspelt word finds nothing.
  • A search reaches the first 200 results only. limit is 1 to 50, and offset plus limit may not pass 200.
  • A question is at most 200 characters.
  • References are not inlined. Text that lives in a component is found under the component's key, not under the page that uses it.
  • A missed event needs a rebuild, because the index has no expiry and does not repair itself.
  • A component cannot exclude one of its fields. Searchable, on a document or a folder, is the only way to keep text out.
  • Search covers published content only, so the route refuses ?preview= and ?at= with 422 Search covers published content only.
  • A rebuild runs at most five times an hour per environment, and the sixth gets a 429.
  • The dashboard has no search box over drafts. This index is for sites, behind a delivery key.

Last updated