Upgrades, migrations and health

Upgrade the CMS to a new image with its migration run first, and read its probes and logs to know it is healthy.

Why indexes are built before the new version answers

A new version of the CMS can ask the database for indexes the old one did not have. An index that arrives late was not there for the writes that came before it. A unique rule that missed a duplicate is worse: it cannot be added afterwards until someone cleans up by hand. That is why the CMS never builds indexes on its own in production. A migration builds them, before the new version takes traffic. The probes and the shutdown sequence serve the same moment: they let a rollout replace pods without dropping the requests those pods were serving.

What the migration, the probes and the drain do

The migration is a command in the image, node apps/api/dist/.scripts/scripts/migrate.js. It reads the same environment as the CMS. For every collection it creates the indexes this version declares, drops the ones it no longer declares, and does nothing when the two already agree. Running it on every deploy is safe, and both the Helm chart and the compose file on Run with Docker use it that way. It writes one JSON line per step to standard output:

• {"event":"index.synced", …} for each collection, with any index it dropped;
• {"event":"environment.reserved", …} for an environment named assets, a name the delivery API now reserves; the line names the one route that environment loses, /d/v1/<project>/assets/list;
• {"event":"environment.orphaned", …} for content left under an environment that no longer exists, which an interrupted clone can leave behind;
• {"event":"migrate.complete"} at the end.

The two warnings report and change nothing, because removing content and renaming an environment are decisions for the project's owner. A failure writes {"event":"migrate.failed", …} to standard error and exits with a non-zero code. That exit code is what stops a rollout.

The two probes ask different questions:

/healthz touches no dependency, on purpose. A pod whose Redis is briefly unreachable should stop taking traffic, not be killed and restarted. /readyz answers with the state of each store, such as {"status":"ok","mongo":"up","redis":"up"}.

The shutdown is graceful. On SIGTERM or SIGINT the process does four things in order: it starts answering /readyz with 503, stops accepting new connections, waits up to SHUTDOWN_GRACE_MS for the requests already in flight, and only then closes Mongo and Redis. Each step writes one log line, from Shutdown started to Shutdown complete. Keep SHUTDOWN_GRACE_MS below your orchestrator's own grace period, or the forced kill arrives in the middle of the drain.

The log is one JSON object per line on standard output, with level, time, requestId, context and message. An ordinary start prints one line that tells you what you need to know: Listening on <BASE_URL> — assets on disk, serving the dashboard. A start that fails on configuration prints Invalid configuration: followed by one line per variable that is wrong. LOG_LEVEL sets how much the CMS writes.

How to

Before an upgrade, read the release notes for the version you are moving to, and take a backup of every project you cannot afford to lose, as Backups describes. Then change the tag and let the migration run before the new CMS starts.

docker compose pull
docker compose up -d --wait cms
docker compose logs migrate

docker pull iegork/quillet:1.1
docker run --rm --env-file quillet.env -v quillet-assets:/data/assets iegork/quillet:1.1 node apps/api/dist/.scripts/scripts/migrate.js

docker stop quillet
docker rm quillet
docker run -d --name quillet --restart unless-stopped --stop-timeout 15 --env-file quillet.env -v quillet-assets:/data/assets -p 3000:3000 iegork/quillet:1.1

helm upgrade quillet ./charts/quillet -f values.yaml --set image.tag=1.1.1 --wait

Then check that the new version is healthy:

curl -s http://localhost:3000/readyz

On a server, use your public address in place of localhost:3000. {"status":"ok","mongo":"up","redis":"up"} means the CMS is taking traffic, and degraded names the store it cannot reach. No answer at all usually means the process did not start. Read its log, and if the first lines say Invalid configuration:, fix each variable they name and start it again. The configuration reference says what each one expects.

On Kubernetes, point the probes at the two paths and at nothing else: liveness at /healthz, readiness at /readyz. If liveness points at /readyz, the kubelet kills pods in the middle of a drain. The chart already sets both probes this way.

Limits

• There is no rollback of the database. Running an older image's migration drops the indexes a newer version added, and the next upgrade's migration builds them again.
• The migration builds indexes and fills in a few derived fields. It does not rewrite your content, and it does not rename or delete anything it reports.
• /healthz does not check Mongo or Redis. A container that is up but cannot reach its stores stays alive and reports itself through /readyz alone.
• The listening socket closes as soon as a drain starts, so a probe that opens a new connection then gets a refused connection rather than the 503. A preStop sleep of a few seconds, as the chart has, gives the load balancer time to notice before that.
• A reader waiting for an image variant waits at most SHUTDOWN_GRACE_MS less two seconds, and never more than 20 seconds. If a replica is replaced mid-transform, readers of that one variant get 503 for up to 30 seconds until its lock expires.
• Upgrade the CMS, the SDK and the CLI separately. An image upgrade changes neither of the packages your sites and scripts use.

Related

• Run with Docker has the compose file whose migrate service runs on every start.
• Kubernetes with Helm wires the migration hook, the probes and the drain into a release.
• Configuration reference explains every message Invalid configuration: can print.
• Backups takes the copy to keep before an upgrade.
• Environments covers creating and removing the environment a reserved name affects.