Stage promotion see history edit this page

Talks about: , , and

A promotion gate decides whether a healthy, applied stage may advance the rollout to the next stage. It is distinct from readyChecks , which decide whether a stage’s just-applied objects became Ready: a promotion gate runs after the stage is Ready and gates only advancement. Holding a gate parks the rollout at the stage — the stage stays applied and its drift keeps being corrected, and later stages are never touched.

Two mechanisms are available, and they compose: a soak window runs first, then a manual gate.

Soak before advancing

Hold at a stage for a fixed duration after it becomes Ready, advancing only if it stays healthy for the whole window. This catches delayed regressions — an OOM after warm-up, error-rate creep, a crashloop after several minutes — that point-in-time readyChecks cannot see.

apiVersion: stages.metio.wtf/v1
kind: StageSet
metadata:
  name: web
  namespace: apps
spec:
  serviceAccountName: deployer
  stages:
    - name: staging
      sourceRef:
        name: web-staging
      promotion:
        soak: 10m
    - name: prod
      sourceRef:
        name: web-prod

While staging soaks, the StageSet reports Ready=False with reason Soaking and status.stages[].promotionState carries the soakUntil instant. The controller requeues at that instant and advances on its own once the window elapses. There is no default soak — omit soak (or set it to 0) for no hold; soaks are a per-stage trade-off, so use a long window for prod and none for dev.

Block on pod restarts

A soak waits, but a bare soak only re-checks readiness — and a Deployment can stay Available while individual pods crash-loop and restart behind it. A restartGate closes that gap with no external dependency: each check watches a group of pods by label and blocks the promotion if their container restarts exceed maxRestarts.

    - name: staging
      sourceRef:
        name: web-staging
      promotion:
        soak: 10m
        restartGate:
          onFailure: Hold        # default for every check below (Hold | Rollback)
          checks:
            - name: api
              selector:
                matchLabels:
                  app: web-api
              maxRestarts: 0     # no restarts tolerated (the default)
            - name: workers
              selector:
                matchLabels:
                  app: web-worker
              maxRestarts: 2     # tolerate a couple of blips
              onFailure: Rollback  # override the gate default for this group
    - name: prod
      sourceRef:
        name: web-prod

Each check sums the restart counts across the init and regular containers of every pod matching its selector, in the StageSet’s namespace, and fails once the total exceeds maxRestarts (0 by default — no restarts allowed). Pods being terminated are skipped, so a prior revision draining out during the rollout doesn’t gate the revision replacing it. Pods are matched by label, not by a workload reference, so a group can span any source — a Deployment, StatefulSet, DaemonSet, Job, or a custom controller. Pair it with a soak so the window gives a crash time to surface; it catches the OOM-after-warm-up or crashloop-after-N-minutes that point-in-time readyChecks miss.

onFailure decides what a breach does — set it once on the gate and override it per check:

While a check is breached the StageSet reports Ready=False with reason PromotionBlocked, naming the failing check and the restart total on status.stages[].promotionState. A manual promotion is break-glass over it.

The watched pods are read under the same identity that applies the stage. In the default single-tenant setup — a StageSet with no serviceAccountName or kubeConfig — that is the controller, and the Helm chart grants it pods read out of the box. A StageSet that sets serviceAccountName reads under that ServiceAccount, so grant it pods list; a kubeConfig target reads under that kubeconfig’s identity.

Block on bad events

A pod can stay Ready and never restart while the API streams Warning events about it — new replicas that can’t schedule, probes flapping under load, an image that won’t pull. An eventGate watches those events on a group of pods and blocks the promotion once they pile up.

    - name: staging
      sourceRef:
        name: web-staging
      promotion:
        soak: 10m
        eventGate:
          onFailure: Hold        # default for every check below (Hold | Rollback)
          checks:
            - name: api
              selector:
                matchLabels:
                  app: web-api
              reasons:            # only these Warning reasons count
                - FailedScheduling
                - OOMKilling
                - FailedMount
                - ErrImagePull
              maxEvents: 0        # any matching event fails (the default)
    - name: prod
      sourceRef:
        name: web-prod

Each check counts Warning events whose reason is in its reasons list and whose target pod matches the selector, in the StageSet’s namespace, and fails once the total (by occurrence count) exceeds maxEvents (0 by default). reasons is required — events are noisy, so a check only counts the reasons you name. Events are matched to the exact pods running this revision, so a previous revision’s events don’t carry over.

selector and onFailure work exactly as they do for the restart gate — pods matched by label (any source), the gate default overridable per check, and Rollback reverting the stage to its last-good revision. While a check is breached the StageSet reports Ready=False with reason PromotionBlocked, naming the failing check and the event total. The apply identity reads events under the same rules as the restart gate — the chart grants the controller events read out of the box; a serviceAccountName or kubeConfig target needs events list (alongside pods list) on that identity.

Require a manual promotion

Hold at a stage until an operator promotes it — the “confirm before prod” gate.

    - name: prod
      sourceRef:
        name: web-prod
      promotion:
        requireManualPromotion: true

While held, the StageSet reports Ready=False with reason AwaitingPromotion. Promote the stage when you are satisfied it is healthy:

stagesetctl promote web --namespace apps --stage prod

promote stamps a one-shot stages.metio.wtf/promote annotation that advances the named stage exactly once; the rollout then continues. Add --wait to block until the controller records the promotion. A manual gate that is always rubber-stamped is theatre — it earns its keep when the operator actually checks the stage’s behavior before promoting. requireManualPromotion defaults to false, and is distinct from a migration’s requireApproval , which gates a destructive version transition rather than a stage advance.

Advance only if metrics stay healthy

An analysis advances the stage only while metric checks against an external source keep passing — error rate, latency, an SLO burn rate, anything Prometheus can answer. This sees behavior a Deployment’s own .status cannot, and is the difference between “did it become Ready” (readyChecks ) and “is it behaving” (analysis).

    - name: staging
      sourceRef:
        name: web-staging
      promotion:
        soak: 10m            # observe across this window
        analysis:
          interval: 1m       # re-evaluate this often while holding; defaults to spec.interval
          failureLimit: 3    # consecutive failing evaluations tolerated; defaults to 0
          checks:
            - name: error-rate
              source:
                prometheus:
                  address: http://prometheus.monitoring:9090
                  query: sum(rate(http_requests_total{stage="staging",code=~"5.."}[1m]))/sum(rate(http_requests_total{stage="staging"}[1m]))
              threshold:
                max: "0.01"  # ≤1% 5xx
            - name: latency-p99
              source:
                prometheus:
                  address: http://prometheus.monitoring:9090
                  query: histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{stage="staging"}[5m])) by (le))
              threshold:
                max: "0.5"   # ≤500ms p99

Every check must stay within its threshold (min and/or max, inclusive). A breach increments a failure counter; the counter resets on a passing evaluation and fails the promotion once it exceeds failureLimit. While analysis is holding, the StageSet reports Ready=False with reason Soaking (during the soak) or, once a check has failed past the limit, PromotionBlocked. Each check’s last observed value and verdict are on status.stages[].promotionState.lastAnalysis. The analysis shares the metric source contract with the error-budget freeze , including its SSRF guard and secretRef bearer-token support.

onFailure decides what a failed analysis does:

onSourceError decides what an unreadable source does, defaulting to Hold — never advance a stage whose behavior can’t be verified. (This is the opposite of the error-budget freeze’s Allow default, because holding a promotion only parks the rollout at the current healthy stage.) See the PromotionBlocked and BudgetSourceUnavailable runbooks. While tuning a new analysis, set dryRun: true to record what would block without holding or rolling back anything.

Promote early when the system is healthy

A long soak is insurance against a slow regression — but when the system is demonstrably healthy there’s no reason to wait it out. fastTrack shortens a soak based on a metric: once a minimum soak has elapsed and a burn-rate (or similar) metric stays within bounds, the stage promotes early.

      promotion:
        soak: 30m              # the maximum soak
        fastTrack:
          after: 5m            # always soak at least this long
          max: "1"             # promote early once burn rate <= 1
          source:
            prometheus:
              address: http://prometheus.monitoring:9090
              query: slo:current_burn_rate:ratio{sloth_service="checkout"}

fastTrack only ever promotes earlier than soak — it never extends it. If the metric is over max, unreadable, or the minimum after hasn’t elapsed, the stage just keeps soaking as normal. Use it with analysis when you also want to block on a bad metric: analysis holds/rolls back on breach, fastTrack accelerates on health. fastTrack requires a soak (there’s nothing to shorten otherwise), and after defaults to 0 (a healthy metric can promote immediately).

Combine soak and a manual gate

With both set, the stage soaks first, then awaits a manual promotion:

      promotion:
        soak: 30m
        requireManualPromotion: true

stagesetctl promote is also a break-glass: promoting a soaking stage ends the soak early and advances it immediately, whichever gate is currently holding it.