Key custody: HSM and KMS

By default the master key that encrypts secrets at rest comes from the environment (MASTER_KEY), so anyone who can read the deployment’s configuration can read every signing key, second-factor secret and SMTP password in a database backup. With a key custody backend an HSM or a key-management service holds the master key instead, and neither the configuration nor a backup is enough to decrypt anything without it.

Each backend is an optional cargo feature, so a plain cargo build stays provider-neutral. The container image is built with all of them; the static (musl) release binaries with all but PKCS#11, because a static binary cannot load the HSM vendor’s library. A build without the backend KEY_WRAPPER names refuses to start and names the feature.

How it works

This is envelope encryption. Every master-key generation is a random 256-bit data key. The backend encrypts (“wraps”) it once, when the generation is created, and only the wrapped form is stored, in the master_key_generations table. Each node asks the backend to unwrap every generation when it starts and keeps the data keys in memory; rows are then encrypted and decrypted locally, exactly as under MASTER_KEY (XChaCha20-Poly1305, or AES-256-GCM under a per-value derived key in the FIPS build; bound to the row, tagged with the generation).

What that means in practice:

• No request waits on the backend. Signing a token, checking a TOTP code or sending a webhook never calls the HSM or KMS, pays for a KMS request or counts against its rate limit. The backend is asked once per generation per node start.
• An outage of the backend does not stop running nodes. A node that cannot reach it at start-up refuses to start (it would otherwise run on the wrong key), so scaling out and restarts wait for the backend.
• The data keys are in the nodes’ memory, as MASTER_KEY always was. Custody protects the key at rest, in backups, in configuration and in CI; it does not make a compromised node’s memory safe.
• Revoking rIDM’s access to the backend key stops every new node from starting. Deleting the backend key destroys every secret still under a generation it wrapped: treat it like losing the master key.

Generations from the environment and from the backend can coexist: the environment’s MASTER_KEY / MASTER_KEY_PREVIOUS stay readable next to wrapped generations, which is how an existing deployment moves onto a backend without downtime. A generation number is never reused, and a wrapped generation is always numbered above every generation a stored row carries.

Settings

PKCS#11

The HSM must allow AES-GCM with a caller-supplied IV for C_Encrypt/C_Decrypt on that key.

AWS KMS

Region and credentials come from the SDK’s standard chain (AWS_REGION, IRSA, EKS Pod Identity, the ECS task role, an instance profile, AWS_PROFILE, SSO). The generation is sent as the encryption context ridm:master-key = ridm:master-key:v<N>, so a key policy can require it and CloudTrail shows which generation a node unwrapped. The least the role needs:

{
  "Effect": "Allow",
  "Action": ["kms:Encrypt", "kms:Decrypt"],
  "Resource": "arn:aws:kms:us-east-1:123456789012:key/…",
  "Condition": { "StringLike": { "kms:EncryptionContext:ridm:master-key": "ridm:master-key:v*" } }
}

KMS automatic key rotation is transparent: older generations keep decrypting.

Vault and OpenBao (Transit)

The policy rIDM needs:

path "transit/encrypt/ridm" { capabilities = ["update"] }
path "transit/decrypt/ridm" { capabilities = ["update"] }

Rotating the Transit key (vault write -f transit/keys/ridm/rotate) is transparent; keep min_decryption_version at or below the oldest version a generation used.

Google Cloud KMS

The service account needs roles/cloudkms.cryptoKeyEncrypterDecrypter on the key. Key rotation in Cloud KMS is transparent.

Azure Key Vault and Managed HSM

Without tenant and client ids and a federated token or secret, rIDM asks the managed identity endpoint, then the instance metadata service. The identity needs the wrapKey and unwrapKey key permissions (the Key Vault Crypto User role). Creating a new key version is transparent as long as the older versions stay enabled.

Starting a new deployment on a backend

Set KEY_WRAPPER and the backend’s settings, and leave MASTER_KEY unset. The first node to start creates generation 1: it has the backend wrap a random data key, checks that the backend gives the same key back, and stores the wrapped form. Every later node unwraps it. The global audit chain records master_key.generation_created.

Moving an existing deployment onto a backend

1. Add the backend to every node, keeping MASTER_KEY (and MASTER_KEY_VERSION, MASTER_KEY_PREVIOUS) as they are, and restart. The first node creates the first wrapped generation, numbered after the environment’s; new secrets are written under it and the old ones still decrypt.
2. Re-encrypt what is still under the environment’s generations: ridm master-key rotate, ridm-api rotate-master-key, or the console’s Re-encrypt pending rows (see Rotating keys).
3. Check with ridm master-key status that no row is left under the environment’s generations.
4. Remove MASTER_KEY (and MASTER_KEY_VERSION, MASTER_KEY_PREVIOUS) from every node and restart. Destroy the old key once you no longer need to read backups taken before step 2.

Going back works the same way the other way round: MASTER_KEY current, KEY_WRAPPER unset, the backend in KEY_WRAPPER_PREVIOUS until the rotation is done. Moving from one backend to another likewise: the new one as KEY_WRAPPER, the old one in KEY_WRAPPER_PREVIOUS.

New generations

A new data key is a new generation:

ridm master-key new-generation     # over the admin API
ridm-api rotate-master-key --new-generation   # with the server's configuration; also re-encrypts

or New generation on the console’s Signing keys page, or POST /admin/master-key/generations (a global administrator with ridm:keys:write). The node that creates it starts encrypting under it at once; the other nodes adopt it within a minute (they look for new generations every minute, and load one on first sight of a row under it). Then re-encrypt as above. Rotating the backend’s own key (KMS automatic rotation, a new Transit or Key Vault key version) needs nothing from rIDM.

With the master key in the environment there is no such command: a generation is rolled out by changing MASTER_KEY and MASTER_KEY_VERSION, and the endpoint answers 409.

Status

ridm master-key status, ridm-api rotate-master-key --status, GET /admin/master-key and the console show the backend, every generation (from the environment or which backend and key wrapped it, and whether this node could unwrap it) and how many rows each table has under each generation.

At start-up a node logs each generation it cannot unwrap. That is fatal only for the backend’s current generation, or when no signing key decrypts (the start-up check); rows under an unreadable older generation fail to decrypt until its backend is configured again.

Backups

A database backup holds only wrapped data keys. To restore one you need the backend key that wrapped them (and, for rows still under the environment’s generations, those master keys). Do not schedule deletion of a KMS key, destroy a Transit key or purge a Key Vault key while master_key_generations or any backup you may restore still names it.

Kubernetes and OpenShift

The Helm chart takes the backend under keyCustody: wrapper, previous, the non-secret settings in env, the secret ones (VAULT_TOKEN, PKCS11_PIN, AZURE_CLIENT_SECRET) from a Secret of yours named in existingSecret, and mountServiceAccountToken: true for Vault’s Kubernetes auth method. Workload identity needs no secret at all: annotate the service account (serviceAccount.annotations, e.g. eks.amazonaws.com/role-arn, iam.gke.io/gcp-service-account, azure.workload.identity/client-id) and add the pod label azure.workload.identity/use: "true" for Azure. With a backend, masterKey may be left empty. The migration Job never receives the master key or the backend’s settings: migrations do not read secrets.