Postmortem — PROD-NA tenant-settings 1.2.0 rollout failure in SAAS-SDP-PLT

Severity: Medium  Status: Resolved (workaround applied)

Summary

A rollout of the tenant-settings chart (v1.2.0) across all PROD-NA tenants repeatedly failed and auto-rolled back to the previous version (1.1.0). The failure was caused by a pre-upgrade migration hook Job being OOMKilled. There was no customer-facing impact — Flux safely reverted each tenant to its last healthy release. The rollout was unblocked with a temporary memory increase (1 GiB), and a permanent code fix is tracked in PLT-15009.

Impact

• Who: All PROD-NA tenants targeted by the 1.2.0 rollout.
• What: The 1.2.0 upgrade could not complete; every tenant’s HelmRelease repeatedly reported UpgradeFailed and rolled back to 1.1.0.
• Customer effect: None. Tenants continued running on the last known-good configuration; no downtime, data loss, or state corruption.
• Duration: From the start of the 1.2.0 rollout until the 1 GiB workaround was applied on 2026-07-07.

Root cause

Chart v1.2.0 introduced a one-time pre-upgrade Helm hook Job, <tenant>-tenant-settings-setup-tf-migrate-tfstate, which renames the Terraform state secret and lock lease to a new naming scheme (required by Terraform 1.10+, which rejects a secret_suffix ending in a digit).

As a safety check, the migration script lists all secrets in the namespace and filters them with jq:

kubectl get secrets -n prod-tenants -o json | jq '…filter by tfstateSecretSuffix label…'

prod-tenants is a shared namespace across all tenants. At the time of the incident it held 627 secrets — 389 of them large Helm-release blobs — so get secrets -o json returned roughly 28 MB of JSON. kubectl buffers that entire response and jq parses the whole document into memory (a working set of several hundred MB at peak), which exceeded the hook’s memory limit and triggered an OOMKill (exit 137). Because the Job’s backoffLimit is 0, a single kill immediately failed the hook, which failed the Helm upgrade, which triggered the automatic rollback.

A first attempt to raise the limit from 128Mi to 256Mi did not resolve it, because the memory demand is not fixed — it scales with the number of secrets in the shared namespace and will grow as tenants and Helm revisions are added.

Detection

• Flux/helm-controller reported UpgradeFailed … pre-upgrade hooks failed … job … failed: BackoffLimitExceeded.
• Pod status showed OOMKilled, exit code 137.

• Confirmed in Grafana/Loki via the Kubernetes events stream:

  {job="kubernetes_events"} |~ "migrate-tfstate" |~ "BackoffLimitExceeded"
  {job="kubernetes_events"} |~ "tenant-settings" |~ "UpgradeFailed"
  

Resolution & workaround

To unblock the rollout immediately, the migration hook Job’s memory was raised to 1 GiB (request == limit) for all tenants. This provides comfortable headroom over the observed ~250–400 MB peak and allowed the migration to complete and the 1.2.0 upgrade to succeed.

resources:
  requests: { cpu: 50m, memory: 1Gi }
  limits:   { memory: 1Gi }

Note: 1 GiB is a stopgap. Because the memory demand scales with the size of the shared namespace, this value would need to grow over time — hence the permanent fix below.

Permanent fix (tracked in PLT-15009)

Replace the “list all secrets + filter in jq” pattern with a server-side label selector, so Kubernetes returns only the one or two matching tfstate secrets instead of all 627:

kubectl get secrets -n "$KUBERNETES_NAMESPACE" \
  -l "tfstateSecretSuffix=$OLD_SECRET_SUFFIX" -o name | wc -l

This reduces the payload from ~28 MB to a few KB and makes the hook’s memory usage constant regardless of tenant count (128Mi would again be sufficient). Optional hardening: restrict with --field-selector type=Opaque to exclude Helm-release secrets, and set backoffLimit: 1 so a transient failure retries instead of failing the release.

Lessons learned

• Namespace-wide list calls don’t scale. Any get <resource> -o json over a shared multi-tenant namespace grows unbounded; always use server-side label/field selectors in hook scripts.
• Bumping memory treated the symptom. When demand scales with environment size, a fixed limit is a temporary fix at best.
• Guardrails worked as designed. Flux’s automatic rollback kept every tenant on a healthy version with zero customer impact.
• Consider backoffLimit > 0 for migration hooks so a single transient kill doesn’t fail the whole release.

Action items