Upgrading see history edit this page

Talks about: , , , and

This page lists the actions required when upgrading stageset-controller. Each release that needs action gets a section below, headed by its calendar version; a release with no section needs no migration — a plain upgrade (bump the chart’s appVersion or pull the new image tag) suffices.

After 2026.7.31184504

Five gates changed. Two now cover ground they were documented to cover and did not, so a rollout they used to let through can stop; two became more forgiving, so a rollout they used to block can proceed; one refuses a configuration it previously accepted and silently stalled on. Nothing needs action before upgrading, but three are worth checking against your specs first.

Image policies: skip no longer disarms other policies

An image named under any matching ImageVerificationPolicy’s skip used to be exempt from every matching policy. A skip now exempts an image only from the policy that declares it; another policy naming the same image under images still verifies it.

Where several cluster-scoped policies overlap, an image that was passing unverified may now be verified — and hold its stage under ImageUnverified if it has no signature. Review overlapping policies before upgrading:

kubectl get imageverificationpolicies -o custom-columns=NAME:.metadata.name,IMAGES:.spec.images,SKIP:.spec.skip

An image every matching policy skips stays exempt, including under --require-image-verification.

Migration ladders: down actions count towards the http allowlist

A ladder from spec.migrationsSourceRef may only use http actions when the controller runs with --allowed-action-hosts. That check previously read only each migration’s actions, so a ladder whose http call sat in down was accepted. It now reads both.

A sourced ladder with an http action in down, on a controller with no allowlist, is refused with InvalidSpec. Set --allowed-action-hosts to the hosts those actions legitimately call.

Rollback verifies images

A rollback re-applies through the image-verification gate, so a revert can no longer land an image the forward apply would have refused. A previous revision whose images no longer verify reports ImageUnverified instead of reverting; the stage holds at its failed state rather than reverting to something unverified.

Restart gates measure since the last promotion

promotion.restartGate compared maxRestarts against the lifetime restart totals of the pods it watches. Pods routinely outlive a rollout, so a workload carrying restarts from weeks earlier failed the gate the next time it shipped — permanently, since a lifetime total never falls.

The gate now counts only restarts observed since the stage was last promoted, recorded in status.stages[].promotionState.restartBaseline. A stage stuck on this reason clears on the next reconcile after upgrading. A stage that legitimately breached still breaches. No spec change is needed.

CEL ready-checks honour inProgress

readyChecks.exprs[].inProgress was accepted and ignored. It now decides what a verify timeout means: an object that is not current but still reports inProgress holds the stage under StageProgressing and is re-verified, instead of failing it — and, with spec.rollbackOnFailure, instead of reverting an install that was going to succeed.

A check that declares no inProgress keeps the old behaviour, so anything matching on StageFailed only sees less where you have written one. Set onTimeout: Rollback on the stage to keep reverting on timeout.

Metric sources apply their guards to redirects

--allowed-action-hosts was checked against the address a metric source names and then not again, so a permitted source could redirect the query to any host the SSRF guard allows. Every hop is now checked, which bounds where a query ends up rather than only where it starts. This covers spec.errorBudget, a stage’s promotion.analysis and promotion.fastTrack, and FleetRollout wave gates.

If you run with an allowlist and a source legitimately redirects — a load balancer sending the query to a regional endpoint, say — add the destination host alongside the entry one.

A query carrying a secretRef bearer token now refuses a redirect to a different host outright, allow-listed or not. Go forwards Authorization to a subdomain of the original, so the allowlist alone could not keep a tenant’s token off a host nobody named. Redirects within the same host still carry it. A source that depended on a cross-host redirect to deliver its token was already only working by that quirk; point the address at the final host instead.

FleetRollout wave gates

A wave gate is now bounded by --allowed-action-hosts, like every other outbound call the controller makes. If you run with an allowlist, add the host your fleet gates query or they will report the source as unavailable and hold.

A gate whose source names a secretRef is now refused with GateUnsupported. That credential could never be resolved — a FleetRollout is cluster-scoped, so there is no namespace to read the Secret from — and the rollout used to stall with nothing on its condition explaining why. Expose the fleet-wide metric without authentication, or move the authenticated query to a StageSet-level source, which has both a namespace and a ServiceAccount to read it under.

2026.7.31184504

A stage that is still coming up no longer reports as a failure, and the controller reconciles several StageSets at once. Only the first item below needs anything from you, and only if you alert or gate on StageFailed.

A slow stage now reports StageProgressing

A stage whose verify wait ran out of time while its objects were still making progress used to report Ready=False / StageFailed. It now reports StageProgressing — its own reason, because a release that is still starting and one that broke need different answers. The first converges on its own; only the second wants a human.

A stage that genuinely failed is unaffected: the verify wait still aborts the moment an object reaches a terminal state, and still reports StageFailed.

Anything matching on StageFailed therefore sees less than it used to. That is usually the point — a slow first install stops paging — but a rule that was counting on it to catch everything now has a gap. Check both places the reason surfaces:

# Alerting and dashboards: the reason is a label on the reconcile counter.
grep -rn 'StageFailed' <your-prometheus-rules-and-dashboards>

# Anything polling the Ready condition (a portal, a fleet gate, a CI wait).
kubectl get stageset --all-namespaces \
  -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,REASON:.status.conditions[?(@.type=="Ready")].reason'

Add StageProgressing where you want “still coming up” treated as an alert, or — more usually — leave it out and let it stop waking people.

The hold is also paced by spec.retryInterval (falling back to spec.interval) rather than by the controller’s internal backoff, so the re-check cadence is now a number you set.

Reconciles run four at a time

A stage’s verify wait blocks its worker for the whole stage timeout, and the controller previously ran one reconcile at a time — so one tenant coming up slowly held every other tenant’s reconcile behind it, for as long as it took. Four run concurrently now.

Nothing to configure. Memory scales with it: four reconciles can each hold a fetched artifact and its rendered manifests, so a controller sized against the old serial behaviour on very large releases may want its limit raised.

2026.7.31092513

How long a stage may take to verify, and what happens when it runs out of time, both changed. Only the first item below needs an audit before upgrading.

readyChecks.timeout now takes effect

stages[].readyChecks.timeout was accepted and then ignored: a stage that declared how long its readiness takes was bounded by spec.timeout instead. It is now the most specific level of the timeout ladder — readyChecks.timeout, then stages[].timeout, then spec.timeout, then the default.

This is the one change that can make a timeout shorter than before. A StageSet written from the previous documentation carries readyChecks.timeout: 5m, which was doing nothing and now caps the verify phase at five minutes, below the new fifteen-minute default. List every stage that carries one:

kubectl get stageset --all-namespaces --output=json \
  | jq -r '.items[] | . as $s | .spec.stages[]?
           | select(.readyChecks.timeout != null)
           | "\($s.metadata.namespace)/\($s.metadata.name) stage \(.name): \(.readyChecks.timeout)"'

Raise or remove the value on any stage whose workload needs longer than it says.

The default stage timeout is 15 minutes

A stage with no timeout at any level now waits 15 minutes instead of 5 before failing. Nothing to do: the verify wait already fails fast when an object reaches a terminal failure, so the extra patience applies only to workloads still making progress — a first install running migrations against an empty database is the usual one.

A verify timeout no longer rolls back

With spec.rollbackOnFailure set, a stage that ran out of time used to have its previous manifests restored. It now halts with the new manifests in place, and the next reconcile re-verifies them. A stage whose objects genuinely failed still rolls back.

Set onTimeout: Rollback — on spec, or on a single stage — to keep the old behaviour. Doing so is worth a thought for anything that migrates a database before it serves: restoring older code over a half-finished migration is what the new default avoids.