Secrets & Bitwarden

The problem Plesk didn't have

In Plesk, passwords lived in Plesk's database. You typed them into a form; the system stored them; nobody else saw them. Simple.

GitOps creates a tension: the desired state of the cluster lives in a Git repo, and that repo includes things like database passwords, API keys, and S3 credentials. If we committed them as plain YAML, anyone with repo read access would see production secrets. Bad.

The solution: Sealed Secrets

Sealed Secrets is a controller that lets us commit encrypted secrets to the repo. Only the cluster holds the decryption key — not even someone with full repo access can read the plaintext.

Workflow:

Only the cluster's sealed-secrets-controller (in the sealed-secrets namespace) has the decryption key. If you lose that key, all sealed secrets in the repo are permanently unrecoverable — Bitwarden is your backup.

Bitwarden is the master store

We use Bitwarden as the human-readable store of truth for passwords. Workflow:

1. Generate or rotate a password in Bitwarden.
2. Copy it into a plaintext YAML file in untracked/secrets/<name>.yaml (gitignored).
3. Run ./untracked/secrets/seal.sh — produces sealed-<name>.yaml.
4. Copy the sealed file into manifests_v1/app-constructs/<app>/ where it's needed.
5. Commit the sealed file. Push. ArgoCD + sealed-secrets-controller does the rest.

The seal.sh script is just a list of seal <filename> calls. Every new secret gets one line added to it.

Scopes: namespace vs cluster

Sealed Secrets have a "scope" field that limits where they can be decrypted:

• Namespace-scoped (seal in seal.sh): can only decrypt in the namespace named in the metadata. Most of our secrets.
• Cluster-wide (cluster_seal): can decrypt in any namespace. Used for shared things like regcred (Docker Hub auth), redis-auth-secret, ecn-bucket-access (S3 creds shared across tenants).

Look at the function used in seal.sh to know which kind a given secret is.

The Bitwarden CLI and automation

Some scripts need a plaintext secret at runtime (e.g. terraform apply needs the Hetzner API token). We don't hard-code these anywhere. Instead:

1. bw-unlock.sh reads credentials from a gitignored file, unlocks the Bitwarden CLI non-interactively, and exports a session token.
2. Downstream scripts (tf.sh, build-autoscaler-config.sh) source that and fetch secrets by name from Bitwarden at the moment they need them.

Nothing sensitive persists to disk, nothing lives in shell history.

When and how to rotate a secret

See the Rotate a secret runbook. The short version:

1. New password in Bitwarden.
2. Update untracked/secrets/<name>.yaml.
3. ./untracked/secrets/seal.sh.
4. Copy the new sealed file over the old one.
5. Commit, push.
6. Reloader restarts any pods that mount the secret automatically.

Why you'd mess with any of this

Honestly, for daily operations you probably won't. The secrets-manager Claude agent handles the full flow end-to-end: you say "add a new API secret for X", it creates the plaintext, seals it, places it, and commits. Delegate to it.

But you should understand the concept well enough to know:

• Why a random YAML file in the repo has encryptedData instead of data — it's sealed.
• Why you sometimes see Secret resources in ArgoCD's "ignoreDifferences" — the sealed-secrets-controller owns creating the real Secret, so ArgoCD intentionally doesn't diff it.
• Why we can paste passwords into committed YAML without panic — they're encrypted.

Further reading

• Official Sealed Secrets README
• Our secrets-manager agent — delegate to it.
• Rotate a secret — the actual runbook.
• Next: Ingress, TLS & Cloudflare