Quillet
Menu
Menu

Kubernetes with Helm

6 min read

Deploy the CMS to a Kubernetes cluster with its Helm chart, against a managed Mongo and a managed Redis.

Why the chart writes the production rules down

A production CMS needs more than a container that starts. The database indexes have to be built before new pods take traffic. The probes have to tell a restart from a drain, a shutdown has to finish the requests already in flight, secrets have to stay out of the configuration, and the ingress must not cut a large upload. The chart writes those rules down once, so every deployment gets them by default instead of rediscovering each one during an incident.

What the chart renders

The chart deploys the same iegork/quillet image you run with Docker. By default Mongo and Redis are external, and the chart points at them with connection strings, because a managed database and a managed cache are what it is built for. For an evaluation install, optional Bitnami subcharts can stand both up inside the release.

A release is made of these parts:

  • A Deployment of replicaCount pods (one by default), or an autoscaler between autoscaling.minReplicas and maxReplicas. Each pod runs as uid 1000 with a read-only root filesystem, a writable emptyDir at /tmp for uploads and image transforms in progress, and a memory limit of 768Mi. That limit leaves room for one transform of a 50-megapixel image, which needs about 200 MB.
  • A ConfigMap built from config, auth and assets, with one environment variable per key. A change to it rolls the pods by itself, through a checksum/config annotation.
  • A Secret built from secret, unless you name an existingSecret of your own. Secrets never reach the ConfigMap.
  • The migration, as a Job that runs before install and before every upgrade. It builds the database indexes. Its backoffLimit is 0 and its deadline is 120 seconds, so a failed or stuck migration stops the release instead of letting new pods reach a database whose indexes do not match.
  • The probes. Liveness is on /healthz, which answers for as long as the process runs, a drain included. Readiness is on /readyz, which fails as soon as a shutdown begins or as soon as Mongo or Redis is unreachable.
  • The drain. A five-second preStop sleep lets the Service stop routing to the pod before its socket closes; the pod then has up to SHUTDOWN_GRACE_MS for requests in flight, all inside a 30-second termination grace period. The chart refuses to render a grace period that does not exceed the other two together.
  • An Ingress, when ingress.enabled is on, with TRUST_PROXY=1 for its one hop and an nginx proxy-body-size derived from config.assetMaxBytes.
  • A PersistentVolumeClaim for uploaded files under the default disk backend.

values.schema.json checks image.repository, config.baseUrl, config.trustProxy, ingress.host when the ingress is on, and the bucket's endpoint and name under s3. A release missing one of them fails at helm install, not later in a crash loop.

How to

The chart lives in the charts/quillet directory of the CMS source. It is not published to a chart repository, so the commands below run from a copy of that source.

  1. Create the Secret with the four values the CMS cannot start without. Keeping them out of your values file keeps them out of your history.
kubectl create secret generic quillet-secrets \
  --from-literal=MONGO_URI='<mongodb+srv://user:password@cluster.example.net/quillet>' \
  --from-literal=REDIS_URL='<rediss://default:password@cache.example.net:25061>' \
  --from-literal=SESSION_SECRET="$(openssl rand -hex 32)" \
  --from-literal=WEBHOOK_SECRET_KEY="$(openssl rand -base64 32)"

MONGO_URI must address a replica set, and every managed Mongo is one. A standalone mongod refuses the transactions the CMS writes with.

  1. Write a values file. This is the whole of a minimal one:
image:
  repository: iegork/quillet
  tag: '1.1.1'
ingress:
  enabled: true
  host: <cms.example.com>
  tls:
    issuer: <letsencrypt-prod>
config:
  baseUrl: https://<cms.example.com>
existingSecret: quillet-secrets

ingress.host and config.baseUrl must name the same host, because the chart derives neither one from the other. tls.issuer names a cert-manager ClusterIssuer and adds its annotation to the ingress.

  1. Fetch the subcharts the chart declares, then install.
helm dependency build ./charts/quillet
helm install quillet ./charts/quillet -f values.yaml --wait

The migration runs first, and the pods start after it. Follow them with kubectl get pods. When the release is ready, curl -s https://<cms.example.com>/readyz prints {"status":"ok","mongo":"up","redis":"up"}.

  1. Open the dashboard at your host and create the first account. Registration is open on a new install. Close it afterwards with config.authRegistration: closed, as Sign-in with GitHub or Google describes.

To run more than one pod, move files to a bucket first. Set assets.backend: s3 and the assets.s3 values, put the two bucket keys in the Secret as ASSET_S3_ACCESS_KEY_ID and ASSET_S3_SECRET_ACCESS_KEY, and only then raise replicaCount or turn on autoscaling. Under disk, the chart refuses more than one replica unless the claim is ReadWriteMany. File storage has the details.

Some variables have no value in the chart, such as ASSET_MIME_ALLOWLIST, ASSET_TRANSFORM_CONCURRENCY, IMPORT_MAX_BYTES or REDIS_KEY_PREFIX. Those go in extraEnv, which reaches both the pods and the migration:

extraEnv:
  - name: IMPORT_MAX_BYTES
    value: 512mb

To try the chart on a throwaway cluster with no managed services, turn on both subcharts and let the chart derive the connection strings:

helm install quillet ./charts/quillet \
  --set image.repository=iegork/quillet --set image.tag=1.1.1 \
  --set config.baseUrl=http://localhost:8080 \
  --set mongodb.enabled=true --set mongodb.auth.rootPassword=<password> --set mongodb.auth.password=<password> \
  --set redis.enabled=true --set redis.auth.password=<password> \
  --set secret.sessionSecret="$(openssl rand -hex 32)" \
  --set secret.webhookSecretKey="$(openssl rand -base64 32)"

Choose passwords without characters that need escaping in a URL, because the chart puts them inside the connection strings.

Limits

  • The chart is not in a chart repository. You install it from a copy of the CMS source, and you upgrade it the same way.
  • The subcharts are for evaluation only. They come with no backups and no monitoring, and their passwords sit in your shell history.
  • disk on block storage that is ReadWriteOnce, such as DigitalOcean's, means exactly one replica and no autoscaling. Production with more than one pod means s3.
  • An existing release left on disk gains a PersistentVolumeClaim on its next upgrade. If you would rather the chart did not create one, set assets.disk.existingClaim to a claim you already have.
  • The ingress body limit is one number for every path, derived from config.assetMaxBytes. Keep that value at or above config.jsonBodyLimit; Helm cannot compare the two spellings for you.
  • The chart adds no NetworkPolicy. Webhooks call whatever URLs owners configure, so a cluster that denies egress by default must allow TCP 443 from the CMS pods.
  • Nothing checks what is inside an existingSecret at install. A missing key, such as a bucket key under s3 or a provider secret, shows up later as a pod that fails to boot.
  • To set ASSET_SIGNING_SECRET from an existingSecret, also set secret.assetSigningSecret to any non-empty value. Otherwise the key stays out of the pods' environment.
  • Webhook signatures carry a timestamp that receivers check within five minutes, so the nodes need correct time.

Last updated