Quillet
Menu
Menu

File storage

7 min read

Choose where the media library keeps its files, on a mounted disk or in an S3-compatible bucket, and set the limits that guard them.

Why files live outside the database

The documents live in Mongo. The images, PDFs and videos content managers upload do not: they are bytes, often large, and they belong in storage built for bytes. A single machine has a disk, so a mounted volume is the simplest place for them. Several replicas need one store they can all reach, and a bucket gives you that. Whichever you choose, the CMS stays the only door to the files. It is the part that scales images, keeps private files private, and refuses a file that is not what it claims to be.

How the CMS stores and serves files

ASSET_BACKEND picks the store once, when the process starts:

  • disk writes under ASSET_DISK_ROOT, normally a mounted volume. Each file goes to a temporary name first and is then renamed into place, so nobody ever reads half a file.
  • s3 writes to any S3-compatible bucket, such as DigitalOcean Spaces, MinIO or AWS S3. The bucket stays private and the CMS is its only reader: it writes objects with no ACL and never hands out a link to the bucket itself.

Before it opens a port, the CMS checks the store it was given. On disk it creates the directory, then writes and removes a probe file; on s3 it asks for the bucket. If the check fails, the CMS refuses to start and says why, so a deployment without usable storage fails at once instead of on its first upload. A store that goes away later makes the file routes answer 503 with Retry-After, while everything else keeps working.

Every byte a reader receives comes from the CMS, at /d/v1/<project>/assets/<id>. That route scales an image on request (?w=640&f=webp), sends an SVG with headers that stop it running script, checks the signed link of a private file, and answers range and conditional requests. A CDN can sit in front of those routes, but never in front of the bucket. Set ASSET_PUBLIC_URL to the CDN's origin and the CMS builds every file URL it hands out on that origin instead of on BASE_URL.

Files are public by default, which means anyone with the URL can read them. A content manager can upload a file as private instead, and then the plain URL answers 404. For a private file the CMS hands out a URL that carries a signature and an expiry, stays valid for six to twelve hours, and is signed with ASSET_SIGNING_SECRET or, when that is blank, with a key derived from SESSION_SECRET. A CDN never caches a private file, so every request for one reaches the CMS.

An upload passes these checks, in order:

  1. Rate: ASSET_UPLOAD_RATE_LIMIT uploads per person or key per ASSET_UPLOAD_RATE_LIMIT_WINDOW_MS, and past that the answer is 429.
  2. Size: an upload larger than ASSET_MAX_BYTES is stopped as it arrives and answered 413.
  3. Type: the CMS reads the type from the file's own bytes, never from its name or from what the browser declared, and the type must be on ASSET_MIME_ALLOWLIST. A renamed program is refused because it is not on the list. So is an SVG with scripts, event handlers or external references.
  4. Image: images over 50 megapixels are refused. A photo's location and other metadata are stripped before it is stored.
  5. Duplicate: the same bytes uploaded twice to one project become one file.
  6. Quota: past ASSET_PROJECT_QUOTA_BYTES for the project, the answer is 409, with how much is used.

The CMS makes a scaled image on its first request and keeps it beside the original. ASSET_IMAGE_WIDTHS lists the widths a caller may ask for, and the formats are webp, avif, jpeg and png, so one image has a bounded set of variants. ASSET_TRANSFORM_CONCURRENCY caps how many images one process scales at once, because each transform of a 50-megapixel image needs about 200 MB of memory.

How to

Start with the store.

Disk

Mount a volume and point ASSET_DISK_ROOT at it. The image runs as the node account (uid 1000), and that account must own the directory; Run with Docker shows the one-time chown.

ASSET_BACKEND=disk
ASSET_DISK_ROOT=/data/assets

With the Helm chart, disk is the default. The chart creates a claim of assets.disk.size (20Gi) and mounts it at /data/assets. More than one replica needs assets.disk.accessMode: ReadWriteMany, and the chart refuses to render any other access mode.

S3-compatible bucket

Create a private bucket and a key that can read and write it, then set the group. This is DigitalOcean Spaces with virtual-hosted addressing:

ASSET_BACKEND=s3
ASSET_S3_ENDPOINT=https://<region>.digitaloceanspaces.com
ASSET_S3_REGION=us-east-1
ASSET_S3_BUCKET=<bucket>
ASSET_S3_ACCESS_KEY_ID=<access key>
ASSET_S3_SECRET_ACCESS_KEY=<secret key>
ASSET_S3_FORCE_PATH_STYLE=false
ASSET_S3_PREFIX=

For MinIO and most self-hosted stores, set ASSET_S3_FORCE_PATH_STYLE=true. Set ASSET_S3_PREFIX to a folder name, such as quillet, when the bucket also holds other things. Under s3, the four variables without a default (endpoint, bucket and both keys) are required together. Under disk, the CMS ignores the whole group.

With the Helm chart, set assets.backend: s3 and assets.s3.endpoint, region, bucket, forcePathStyle and prefix in your values. Put the two keys either in secret (s3AccessKeyId, s3SecretAccessKey) or as ASSET_S3_ACCESS_KEY_ID and ASSET_S3_SECRET_ACCESS_KEY in your existingSecret. A bucket is what lets the chart run more than one replica.

Restart the CMS and read its first log line, which names the store it is using:

docker compose logs cms | grep Listening

The message ends in assets on disk, serving the dashboard or assets on s3, serving the dashboard. With docker run, read the same line with docker logs quillet.

Then make whichever of these changes apply to you:

  • A CDN: point a pull zone at the CMS's own address, not at the bucket, and set ASSET_PUBLIC_URL to the zone's origin, such as https://cdn.example.com, without a trailing slash.
  • Links to private files: set ASSET_SIGNING_SECRET to 32 or more random characters (openssl rand -base64 48) if you want to be able to cancel every outstanding private link without signing everyone out. Changing it cancels them all at once.
  • Larger uploads: raise ASSET_MAX_BYTES, and raise the body limit of every proxy in front of the CMS to match, or the proxy cuts uploads at the edge.
  • More room: raise ASSET_PROJECT_QUOTA_BYTES, or set 0 for no quota.
  • Another file type: add its media type to ASSET_MIME_ALLOWLIST, keeping the defaults you still want, for example …,font/woff2,font/woff.
  • Other image widths: set ASSET_IMAGE_WIDTHS to the widths your layout's srcset asks for.
  • More transforms at once: raise ASSET_TRANSFORM_CONCURRENCY only together with the container's memory limit.

Limits

  • One store per deployment. Files belong to a project, not to an environment, and every project on the instance uses the same backend.
  • Changing ASSET_BACKEND, ASSET_DISK_ROOT or ASSET_S3_PREFIX does not move files that are already stored. The CMS reads only from the store it started with, and files it cannot find answer 404.
  • disk with more than one replica needs a volume they all mount. Two replicas on two separate volumes each answer 404 for the other's files.
  • Uploads and transforms pass through the container's temporary directory, so a read-only root filesystem needs a writable /tmp, as the Helm chart mounts.
  • A signed link to a private file is a bearer credential until it expires, which can be up to twelve hours. Anyone who obtains one can read that file until then.
  • Making a public file private stops its plain URL at once, but copies a CDN or a browser already cached may stay readable for up to a year. Upload a file as private from the start when that matters.
  • TIFF images, if you add them to the allow-list, are stored but never scaled. HEIC photos are refused even if you add them, because the image library cannot read them.
  • Animated GIFs and SVGs are served as uploaded, and the CMS ignores a width or format request on them.
  • Images larger than 50 megapixels are refused, whatever ASSET_MAX_BYTES allows.
  • Files and media explains why files are project-wide, what public and private mean, and how variants work.
  • Media shows content managers how to upload, preview and mark files private.
  • Assets builds image URLs, srcset and signed links from your site's code.
  • Configuration reference has every asset variable, with what breaks when each one is wrong.
  • Behind a proxy and TLS sets the body limits at the edge.

Last updated