Ecommercen Ops

Appearance

Rotate a secret ​

This page walks through rotating any secret the cluster consumes — database passwords, API tokens, registry credentials, Cloudflare tokens, whatever. The mechanics are the same for all of them; only the target files differ.

When you rotate ​

  • A credential leaked (or you strongly suspect it did — e.g. it showed up in a log, a pastebin, a screen-share).
  • A colleague who had access left the team.
  • You're doing a routine rotation because something is old. We don't run on a strict schedule, but anything more than a year old is a fair candidate.
  • Bitnami rotated the sealed-secrets controller key (rare — you'd know, because many apps would start failing at once).

For 99% of operators, reaching for this page happens once or twice a year.

The flow ​

1. Update the source of truth (Bitwarden) ​

Rotate the password at the service that owns it — e.g. Docker Hub for regcred, Cloudflare for the API token, ALTER USER on MariaDB for a DB password. Copy the new value into the corresponding Bitwarden item.

2. Edit the plaintext YAML ​

Every secret the cluster uses has a plaintext Kubernetes Secret in untracked/secrets/<name>.yaml. Open it, paste the new value.

  • 🟢 This file is gitignored — the whole untracked/ tree is. Plaintext never leaves your machine.
  • 🟠 Double-check you're editing untracked/secrets/<name>.yaml, not something under manifests_v1/ — committing plaintext is the one mistake to avoid.
  • 🔴 Never share the file over Chat / email / screenshots. Bitwarden is the sharing channel.

3. Seal it ​

From the repo root:

bash
cd untracked/secrets
./seal.sh

seal.sh is a short shell script with one seal <file> or cluster_seal <file> line per secret we manage. It shells out to kubeseal, which contacts the in-cluster sealed-secrets-controller to fetch the public key, then writes the encrypted form to sealed-<name>.yaml alongside the plaintext. The scope matters:

  • seal — namespace-scoped. The sealed file can only decrypt in the namespace named in its metadata.
  • cluster_seal — cluster-wide. Any namespace can decrypt it. Used for things like regcred that multiple tenants share.

4. Copy the sealed file to its committed location ​

The sealed output stays in untracked/ by default; you have to move it into place. Mapping examples:

SecretDestination
regcredmanifests_v1/app-constructs/ecommercen-clients/wecare/adveshop4/base/sealed-regcred.yaml (and two other bases — it's cluster-wide)
tunnel-credentialsmanifests_v1/app-constructs/cloudflared/sealed-tunnel-credentials.yaml
certmanager-cloudflare-tokenmanifests_v1/app-constructs/cert-manager/setup/issuers/sealed-certmanager-cloudflare-token.yaml
wecare/app-db-secretsmanifests_v1/app-constructs/ecommercen-clients/wecare/adveshop4/base/sealed-app-db-secrets.yaml

Shared cluster-wide secrets (regcred, redis-auth-secret, ecn-bucket-access) live in more than one place — the full mapping is documented in the secrets-manager agent's knowledge base. Forgetting one of the copies shows up as an ImagePullBackOff or an auth error in a single tenant.

5. Commit and push ​

bash
git add manifests_v1/app-constructs/<app>/sealed-<name>.yaml
git commit -m "[App: <app>] Rotate <secret-name>"
git push

ArgoCD syncs the SealedSecret resource within a minute. The sealed-secrets controller decrypts it into a real Kubernetes Secret. Stakater Reloader notices the Secret's resourceVersion changed and rolling-restarts every Deployment / StatefulSet that references it. No manual kubectl rollout restart needed.

Worked example — rotating regcred ​

regcred is our Docker Hub pull token. It's cluster-wide and referenced in every tenant's imagePullSecrets.

bash
# 1. Generate a new token on Docker Hub, update the Bitwarden entry "ecnv4 - dockerhub regcred"
# 2. Update the plaintext
$EDITOR untracked/secrets/regcred.yaml       # paste new .dockerconfigjson value

# 3. Seal
cd untracked/secrets && ./seal.sh

# 4. Copy to all three locations (cluster-wide secret used by multiple tenants)
cp sealed-regcred.yaml ../../manifests_v1/app-constructs/ecommercen/bases/adveshop4/
cp sealed-regcred.yaml ../../manifests_v1/app-constructs/ecommercen-saas/base/
cp sealed-regcred.yaml ../../manifests_v1/app-constructs/ecommercen-clients/wecare/adveshop4/base/

# 5. Commit + push
cd ../..
git add manifests_v1/app-constructs/**/sealed-regcred.yaml
git commit -m "[Infra] Rotate regcred Docker Hub token"
git push

Special cases ​

  • 🟠 argocd-secret is patched imperatively in-cluster, not via Sealed Secrets. ArgoCD's server config owns the admin password, and sealing it creates a reconcile loop with ArgoCD's own controller. If you need to rotate it, ask the argocd-manager agent — it knows the kubectl patch path.
  • 🟠 Autoscaler config goes through untracked/secrets/build-autoscaler-config.sh, not seal.sh. It pulls the RKE2 join token from Bitwarden, assembles the per-pool JSON, and seals directly into the manifest tree. Requires the Bitwarden CLI unlocked.
  • 🟢 Sealed output differs every time even if plaintext is unchanged. kubeseal adds a random nonce. That's fine — the committed file just gets a new encrypted blob; decrypted value is the same.
  • 🔴 Never re-key the controller without a full re-seal plan. The old sealed files become unreadable. If this ever comes up, the secrets-manager agent handles it.

The easy path: delegate to the agent ​

The secrets-manager agent owns this flow end-to-end. You tell it what you want rotated; it edits the plaintext, runs the seal, copies to all the right places, and hands back a ready-to-commit diff. Unless you want the manual practice, just say something like:

"Rotate the cloudflared tunnel credentials."

and let it drive.

Further reading ​

Internal documentation — Advisable only