Fleet rollout

When you run one product across many StageSets — one per tenant, often one per
namespace — a new version reaches every tenant at once. A FleetRollout paces that:
it rolls a version across a selected set of StageSets in ordered waves, opening
the next wave only after the current one reaches the version, stays healthy through a
soak and a metric gate, and halts the whole fleet the moment a wave regresses.
A FleetRollout is cluster-scoped — a fleet spans namespaces. It approves the
version each member’s own source already offers; it does not push versions, so it
composes with GitOps rather than bypassing it. By default it does not even name the
version — it derives each member’s target from the advance that member is already
holding, so one FleetRollout becomes a standing wave policy that paces every future
version its members’ sources publish, with nothing to keep in sync.
Make members fleet-managed
A member holds each version advance for the fleet to approve. Set
approvalMode: Always
on every StageSet in the fleet:
apiVersion: stages.metio.wtf/v1
kind: StageSet
metadata:
name: app
namespace: moodle-acme
labels:
app: moodle # the fleet selects on this
ring: "1" # the wave this tenant belongs to
spec:
version:
fromObject: { stage: app }
approvalMode: Always
allowDowngrade: true # only needed if the fleet may roll back (see below)
# …
Without approvalMode: Always a member advances on its own and the fleet cannot
pace it.
Define the rollout
Label your tenants into rings, then roll wave by wave:
apiVersion: stages.metio.wtf/v1
kind: FleetRollout
metadata:
name: moodle
spec:
# No targetVersion: the fleet derives it from each member's own held advance.
selector:
matchLabels: { app: moodle } # the whole fleet
namespaceSelector: # optional; omit to span every namespace
matchLabels: { tenant: "true" }
waves:
- name: canary
selector: { matchLabels: { ring: "0" } }
soak: 30m
gate: # health check before the next wave opens
source:
prometheus:
address: http://prometheus.monitoring:9090
query: 'sum(rate(http_requests_total{app="moodle",code=~"5.."}[5m]))'
threshold: { max: "0.01" }
- name: broad
selector: { matchLabels: { ring: "1" } }
soak: 1h
- name: rest
selector: { matchLabels: { ring: "2" } }
Each wave is a label selector over the members. A member that matches selector but
no wave fails the rollout closed (MembersUnassigned) rather than being silently
skipped.
A wave gate queries an endpoint the controller reaches directly, so the hosts it
may reach are bounded by --allowed-action-hosts exactly as an
error-budget source
is. The gate cannot carry a
secretRef bearer token: a FleetRollout is cluster-scoped, so there is no
namespace to resolve that Secret in, and a gate naming one is refused with
GateUnsupported. Expose the fleet-wide metric without authentication, or put the
authenticated query on a StageSet-level source, which has a namespace and a
ServiceAccount to read it under.
Deriving vs. pinning the version. With no targetVersion (above), the fleet reads
each held member’s status.pendingVersion — the advance its own source is offering —
and approves that, so you never restate a version the source already declares and
the two can’t drift. Set targetVersion: "2.0.0" only to pin a specific, bounded
rollout of exactly that version; a member whose source does not offer the pinned
version waits until it does.
How a rollout progresses
The controller opens one wave at a time:
- Open the wave — stamp the version onto each member’s approval, releasing its held advance. Members reach the version when their own source offers it.
- Settle — wait until every member of the wave has adopted its version and is Ready.
- Soak — hold the wave’s
soakafter it settles. - Gate — evaluate the wave’s
gatemetric. A scalar within the threshold passes; one outside it halts the fleet. - Advance — open the next wave. When every wave passes, the rollout is
Completed.
stagesetctl fleet <name> shows exactly where the rollout is — see
fleet
.
Pin a pilot instance first
To require one specific “smoke-test” tenant to update before anything else, give it its own ring and make it wave 0:
waves:
- name: pilot
selector: { matchLabels: { ring: canary } } # only the test tenant carries this
soak: 15m
gate: { … } # only proceed if it's healthy
- name: fleet
selector: { matchLabels: { ring: prod } }
The pilot is the sole member of wave 0, so the rest of the fleet waits until it reaches the version, soaks, and passes its gate.
Halt on regression
A wave halts the whole fleet — no further waves open — when either its gate
metric falls outside the threshold, or a member that had reached the version goes
not-Ready (a regression). The rollout reports phase: Halted and Ready=False with
reason Halted, and emits a Warning event naming the cause. A halt is re-derived
each reconcile, so it clears on its own once the cause does — a member recovers, or
the gate passes again. A metric the controller cannot read holds the wave (neither
advancing nor halting) rather than guessing.
Roll back a regressed wave
By default a regression only halts. Set onRegression: Rollback with a
previousVersion to also revert the affected wave:
spec:
targetVersion: "2.0.0"
previousVersion: "1.0.0" # required for Rollback
onRegression: Rollback
# …
On a regression the fleet directs the halted wave’s members back to previousVersion
— it stamps a rollback-to annotation that overrides each member’s source-resolved
version with the lower one, unwinding it through the migration
down actions
. A member only actually
reverts if it permits downgrades (spec.version.allowDowngrade) and the crossed
migrations declare down actions; otherwise it refuses, and the fleet stays halted
and says so. Unlike a manifest-only rollback, a member either unwinds its schema
cleanly or refuses — never leaves the schema ahead of the code.
One rollout per StageSet
A StageSet must be governed by at most one FleetRollout — two would both stamp its
approval and fight over it. When a member is selected by more than one rollout, the
rollout fails closed with Ready=False, reason MembersContested, naming the other
rollout, and stamps nothing until the overlap is resolved.
Status reasons
The rollout’s Ready condition carries one of:
| Reason | Meaning | What to do |
|---|---|---|
Progressing | A wave is open and advancing. | Nothing — watch with stagesetctl fleet. |
Completed | Every wave reached the target version. | Nothing. |
Halted | A gate failed or a member regressed. | Investigate the named wave; the halt clears when the cause does. |
MembersUnassigned | A selected StageSet matches no wave. | Add the missing ring label, or a wave that selects it. |
MembersContested | A member is also in another rollout. | Narrow one rollout’s selector so each member has one owner. |
InvalidSelector | A selector is malformed. | Fix the selector. |
A member held awaiting its wave reports its own Ready=False with reason
AwaitingApproval
— that is the normal held state, not
an error.