Fleet rollout see history edit this page

Talks about: , , , , , and

When you run one product across many StageSets — one per tenant, often one per namespace — a new version reaches every tenant at once. A FleetRollout paces that: it rolls a version across a selected set of StageSets in ordered waves, opening the next wave only after the current one reaches the version, stays healthy through a soak and a metric gate, and halts the whole fleet the moment a wave regresses.

A FleetRollout is cluster-scoped — a fleet spans namespaces. It approves the version each member’s own source already offers; it does not push versions, so it composes with GitOps rather than bypassing it. By default it does not even name the version — it derives each member’s target from the advance that member is already holding, so one FleetRollout becomes a standing wave policy that paces every future version its members’ sources publish, with nothing to keep in sync.

Make members fleet-managed

A member holds each version advance for the fleet to approve. Set approvalMode: Always on every StageSet in the fleet:

apiVersion: stages.metio.wtf/v1
kind: StageSet
metadata:
  name: app
  namespace: moodle-acme
  labels:
    app: moodle          # the fleet selects on this
    ring: "1"            # the wave this tenant belongs to
spec:
  version:
    fromObject: { stage: app }
    approvalMode: Always
    allowDowngrade: true # only needed if the fleet may roll back (see below)
  # …

Without approvalMode: Always a member advances on its own and the fleet cannot pace it.

Define the rollout

Label your tenants into rings, then roll wave by wave:

apiVersion: stages.metio.wtf/v1
kind: FleetRollout
metadata:
  name: moodle
spec:
  # No targetVersion: the fleet derives it from each member's own held advance.
  selector:
    matchLabels: { app: moodle }          # the whole fleet
  namespaceSelector:                       # optional; omit to span every namespace
    matchLabels: { tenant: "true" }
  waves:
    - name: canary
      selector: { matchLabels: { ring: "0" } }
      soak: 30m
      gate:                                # health check before the next wave opens
        source:
          prometheus:
            address: http://prometheus.monitoring:9090
            query: 'sum(rate(http_requests_total{app="moodle",code=~"5.."}[5m]))'
        threshold: { max: "0.01" }
    - name: broad
      selector: { matchLabels: { ring: "1" } }
      soak: 1h
    - name: rest
      selector: { matchLabels: { ring: "2" } }

Each wave is a label selector over the members. A member that matches selector but no wave fails the rollout closed (MembersUnassigned) rather than being silently skipped.

Deriving vs. pinning the version. With no targetVersion (above), the fleet reads each held member’s status.pendingVersion — the advance its own source is offering — and approves that, so you never restate a version the source already declares and the two can’t drift. Set targetVersion: "2.0.0" only to pin a specific, bounded rollout of exactly that version; a member whose source does not offer the pinned version waits until it does.

How a rollout progresses

The controller opens one wave at a time:

  1. Open the wave — stamp the version onto each member’s approval, releasing its held advance. Members reach the version when their own source offers it.
  2. Settle — wait until every member of the wave has adopted its version and is Ready.
  3. Soak — hold the wave’s soak after it settles.
  4. Gate — evaluate the wave’s gate metric. A scalar within the threshold passes; one outside it halts the fleet.
  5. Advance — open the next wave. When every wave passes, the rollout is Completed.

stagesetctl fleet <name> shows exactly where the rollout is — see fleet .

Pin a pilot instance first

To require one specific “smoke-test” tenant to update before anything else, give it its own ring and make it wave 0:

  waves:
    - name: pilot
      selector: { matchLabels: { ring: canary } }   # only the test tenant carries this
      soak: 15m
      gate: { … }                                    # only proceed if it's healthy
    - name: fleet
      selector: { matchLabels: { ring: prod } }

The pilot is the sole member of wave 0, so the rest of the fleet waits until it reaches the version, soaks, and passes its gate.

Halt on regression

A wave halts the whole fleet — no further waves open — when either its gate metric falls outside the threshold, or a member that had reached the version goes not-Ready (a regression). The rollout reports phase: Halted and Ready=False with reason Halted, and emits a Warning event naming the cause. A halt is re-derived each reconcile, so it clears on its own once the cause does — a member recovers, or the gate passes again. A metric the controller cannot read holds the wave (neither advancing nor halting) rather than guessing.

Roll back a regressed wave

By default a regression only halts. Set onRegression: Rollback with a previousVersion to also revert the affected wave:

spec:
  targetVersion: "2.0.0"
  previousVersion: "1.0.0"     # required for Rollback
  onRegression: Rollback
  # …

On a regression the fleet directs the halted wave’s members back to previousVersion — it stamps a rollback-to annotation that overrides each member’s source-resolved version with the lower one, unwinding it through the migration down actions . A member only actually reverts if it permits downgrades (spec.version.allowDowngrade) and the crossed migrations declare down actions; otherwise it refuses, and the fleet stays halted and says so. Unlike a manifest-only rollback, a member either unwinds its schema cleanly or refuses — never leaves the schema ahead of the code.

One rollout per StageSet

A StageSet must be governed by at most one FleetRollout — two would both stamp its approval and fight over it. When a member is selected by more than one rollout, the rollout fails closed with Ready=False, reason MembersContested, naming the other rollout, and stamps nothing until the overlap is resolved.

Status reasons

The rollout’s Ready condition carries one of:

ReasonMeaningWhat to do
ProgressingA wave is open and advancing.Nothing — watch with stagesetctl fleet.
CompletedEvery wave reached the target version.Nothing.
HaltedA gate failed or a member regressed.Investigate the named wave; the halt clears when the cause does.
MembersUnassignedA selected StageSet matches no wave.Add the missing ring label, or a wave that selects it.
MembersContestedA member is also in another rollout.Narrow one rollout’s selector so each member has one owner.
InvalidSelectorA selector is malformed.Fix the selector.

A member held awaiting its wave reports its own Ready=False with reason AwaitingApproval — that is the normal held state, not an error.