StageSet Controller

Actions

Mon, 01 Jan 0001 00:00:00 +0000

Actions are typed steps the controller runs around a stage’s apply. They turn an ordered apply into an orchestrated rollout — run a migration before the app, gate the stage on an external check, clean up on failure.

A stage has three action hooks:

pre — run before the manifests are built and applied. A failure aborts the stage with nothing applied.
post — run after the apply is verified. The stage is Ready only if these all succeed.
onFailure — best-effort steps run on any failure from the apply onward.

Each action has a name, optional timeout and retries, and exactly one operation type (patch, http, wait, job, delete, or apply) — enforced by the validating admission webhook. Actions within a hook run in list order.

ArtifactNotFound

Mon, 01 Jan 0001 00:00:00 +0000

Symptom

READY=False, REASON=ArtifactNotFound. Transient: the controller requeues in case the artifact appears.

Cause

A stage’s sourceRef resolves to no ExternalArtifact. Either:

a direct sourceRef (kind: ExternalArtifact, the default) names an object that does not exist in the target namespace; or
a producer sourceRef (e.g. kind: JsonnetSnippet) exists, but no ExternalArtifact carries a spec.sourceRef back-pointer to it yet — the producer has not created its artifact object.

Diagnosis

kubectl describe stageset <name> -n <namespace> # Message names the missing ref
kubectl get externalartifact -n <namespace>

For a producer ref, confirm the producer object exists and that it is configured to publish an ExternalArtifact (not only serve over HTTP):

Building and testing

Mon, 01 Jan 0001 00:00:00 +0000

The controller is a standard Go module. With a Go toolchain installed:

go build ./...
go test -race -cover ./...

Test layers

Unit tests sit next to the code across internal/... and api/v1/. Several are drift gates — e.g. conditions_test.go asserts every Ready Reason has a matching runbook page under docs/content/runbooks/.
envtest-backed tests (envtest_*_test.go) boot a real kube-apiserver + etcd via controller-runtime’s envtest. They t.Skip unless KUBEBUILDER_ASSETS points at an asset bundle — install it with setup-envtest.
Fuzz tests (FuzzXxx) harden the parsing-heavy paths; their seed corpus runs as ordinary unit tests, and -fuzz fuzzes for real.
Kind smoke scenarios under hack/smoke/ run the controller end to end against a real kind cluster.

Static analysis

A pull request must be clean under each of these — run them locally before pushing:

CI and releases

Mon, 01 Jan 0001 00:00:00 +0000

Continuous integration

Every pull request runs verify.yml, which fans out into one job per concern so a failure points straight at the cause:

test — go build then the full go test suite.
lint-go — go vet, staticcheck, gosec, and a gofumpt formatting check.
vulnerabilities — govulncheck (a reachable advisory is a hard gate).
architecture — arch-go against arch-go.yml.
reuse — SPDX/REUSE compliance on every file.
text linters — yamllint, actionlint, markdownlint, typos.
container-image — a buildx image build plus a Trivy scan.

A single all-green job depends on every other job and is the only required check, so new jobs are covered automatically. A separate kind-smoke.yml runs the operator end to end against a real kind cluster, and fuzz.yml exercises the fuzz targets.

Conflict policies

Mon, 01 Jan 0001 00:00:00 +0000

Conflict policies decide what happens when an apply hits an immutable-field conflict — a changed clusterIP, a Job pod template, a StorageClass field that can’t be updated in place. By default the controller fails the stage and reports it, so nothing destructive happens by surprise. A policy opts specific resources into automatic resolution.

The three actions

Fail — stop and report (the default; safest).
Recreate — delete and re-create the object to get past an immutable-field change.
KeepExisting — leave the live object as-is and move on.

A default for the whole stage

spec:
 stages:
 - name: app
 sourceRef:
 name: my-app
 conflictPolicy:
 default: Fail  # explicit; the safe default

The force: true shorthand on a stage is equivalent to conflictPolicy.default: Recreate.

Controller pod down

Mon, 01 Jan 0001 00:00:00 +0000

Symptom

A stageset-controller pod is NotReady; the StageSetControllerPodDown alert fires. While no replica is Ready, StageSets are not reconciled and the Kubernetes admission webhook may reject StageSet writes (failurePolicy: Fail).

Cause

a crash-looping container (bad config flag, missing RBAC, panic),
the node draining or out of resources,
a failing readiness probe (/readyz on --health-probe-bind-address),
the leader-election lease unobtainable.

Diagnosis

kubectl -n stageset-system get pods -l app.kubernetes.io/name=stageset-controller
kubectl -n stageset-system describe pod <pod>
kubectl -n stageset-system logs <pod> --previous --tail=200

Look for flag-parse errors at startup, RBAC Forbidden on the controller’s own ServiceAccount, or OOMKills.

DependencyNotReady

Mon, 01 Jan 0001 00:00:00 +0000

Symptom

READY=False, REASON=DependencyNotReady. Transient: the controller requeues at spec.retryInterval (or spec.interval).

Cause

A StageSet listed in spec.dependsOn is not Ready at its observed generation, so this StageSet holds before doing any work. Semantics match kustomize-controller: a dependency is satisfied only when its Ready=True and its status.observedGeneration equals its current generation (so a freshly-edited dependency mid-reconcile does not count as ready).

Diagnosis

kubectl describe stageset <name> -n <namespace> # Message names the dependency
kubectl get stageset <dependency> -n <namespace> # is it Ready?
kubectl describe stageset <dependency> -n <namespace> # why not?

Remediation

Resolve the dependency’s own Ready condition first (follow its runbook). Once it reports Ready=True at its current generation, this StageSet proceeds on the next reconcile. If the dependency is intentionally suspended, this StageSet waits indefinitely by design — remove the dependsOn entry or resume the dependency.

DowngradeRequiresMigration

Mon, 01 Jan 0001 00:00:00 +0000

Symptom

READY=False, REASON=DowngradeRequiresMigration. Terminal: the run does not requeue until the desired version is at or above status.version.

Cause

The desired version (spec.version) is lower than the version the controller last recorded as deployed (status.version). Downgrades are refused by default: migrations are forward-only action ladders, and replaying upgrade migrations in reverse is how data gets destroyed. The controller does not silently run a downgrade.

Diagnosis

kubectl describe stageset <name> -n <namespace>
kubectl get stageset <name> -n <namespace> -o jsonpath='{.status.version}' # deployed
# desired: read spec.version.value, or the version file the artifact carries

Remediation

Pick the intended direction:

From Jsonnet to a gated rollout

Mon, 01 Jan 0001 00:00:00 +0000

This tutorial follows a complete delivery: write Kubernetes manifests in Jsonnet and publish the source through Flux; JaaS renders it into a Flux ExternalArtifact, and a StageSet rolls it out with a readiness gate.

The chain is:

Jsonnet in Git/OCI/Bucket → JaaS (JsonnetSnippet) → ExternalArtifact → StageSet

This tutorial renders Jsonnet, so it goes through JaaS: JaaS turns the Jsonnet into an ExternalArtifact the stage consumes. (If your manifests were already plain YAML, a stage could read a GitRepository/OCIRepository/Bucket directly — see Stage sources. The renderer is here because the input is Jsonnet, not because StageSet can’t read Git.)

Install on Kubernetes

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

A Kubernetes cluster with kubectl and helm configured against it.
Flux source-controller, specifically the ExternalArtifact API (source.toolkit.fluxcd.io). A StageSet stage always resolves to an ExternalArtifact, so the CRD must exist. ExternalArtifact lands in Flux v2.7.0; install at least that version. The controller also watches GitRepository, OCIRepository, and Bucket sources for producer-aware resolution.
cert-manager, only if you choose the cert-manager webhook certificate mode. The chart defaults to self-signed, which provisions and rotates the admission webhook’s TLS in-process and needs no cert-manager. See production for the trade-off.

JaaS, JOI, or any particular artifact producer are not required to install the controller — those are sources of ExternalArtifacts, wired up per StageSet.

InvalidSpec

Mon, 01 Jan 0001 00:00:00 +0000

Symptom

READY=False, REASON=InvalidSpec. The Message names the offending field or action. Terminal: the controller does not requeue until the spec changes.

Cause

The spec failed validation that the CRD schema cannot express cheaply, normally one of:

an action sets zero or more than one verb — each action must set exactly one of patch, http, wait, job, delete, apply (see actions);
spec.migrations without spec.version, or a migration anchored to a stage name that does not exist (see versioned migrations);
spec.version does not name exactly one source — set one of value, fromObject, or fromArtifact;
spec.decryption.provider is not sops, or a secretRef is given without a name (see encryption);
an invalid update window — a malformed schedule, duration, or timeZone (see update windows).

The admission webhook normally rejects these at write time; seeing this on the object means the webhook was bypassed or disabled and the reconciler caught it.

InvalidVersion

Mon, 01 Jan 0001 00:00:00 +0000

Symptom

READY=False, REASON=InvalidVersion. Terminal: the run does not requeue until the spec or the version file is fixed.

Cause

A version spec.version (or a migration boundary) could not be resolved to a parseable semver. The controller refuses to proceed rather than deploy a half-versioned system — a system whose recorded version is unknown is worse for migrations than an unversioned one. The Message names which input failed. By version source:

spec.version.value — the inline string is not a semver.
spec.version.fromObject — the named stage doesn’t exist; the object (kind/name) isn’t in the stage’s rendered manifests; the fieldPath is invalid JSONPath or resolves to empty; or the value read (by default the app.kubernetes.io/version label) is missing or not a semver.
spec.version.fromArtifact — the named stage doesn’t exist; the file at path is missing from the stage’s artifact, empty, or doesn’t parse as a semver.
spec.version sets none of value/fromObject/fromArtifact.
A migration’s to or from is not a valid semver.
The recorded status.version is not a semver (corrupted status).

Common triggers across all of them: a v prefix or trailing whitespace the parser rejects, or non-semver text (e.g. a Git SHA or a latest tag) where a version was expected.

Multi-cluster and tenancy

Mon, 01 Jan 0001 00:00:00 +0000

There are two ways to run the controller, and they map onto two different trust models. Pick the one that matches your cluster:

Multi-tenant — the controller holds no write access of its own and applies every StageSet impersonating that StageSet’s serviceAccountName. Each tenant’s RBAC bounds what its releases can touch. This is the chart default.
Single-tenant — the cluster has one operator, so per-tenant isolation buys nothing. Run the controller under its own identity bound to cluster-admin and skip impersonation entirely — the model Flux’s helm-controller uses in its default install.

The two sections below set each one up. The optional watch scoping narrows which namespaces a multi-tenant controller sees.

Operations

Mon, 01 Jan 0001 00:00:00 +0000

Metrics

The controller registers custom metrics on the controller-runtime registry, served on --metrics-bind-address (:8080) alongside the standard controller_runtime_* and workqueue_* series. Enable scraping with the chart’s opt-in ServiceMonitor (metrics.serviceMonitor.enabled):

# values.yaml
metrics:
 serviceMonitor:
 enabled: true # needs the Prometheus operator CRDs

Metric	Type	Labels	Meaning
`stageset_reconcile_total`	counter	`namespace`, `name`, `reason`	Reconciles, by terminal Ready reason.
`stageset_stage_applied_total`	counter	`namespace`, `name`, `stage`	Stages applied and verified.
`stageset_drift_corrected_total`	counter	`namespace`, `name`, `stage`	Out-of-band drift re-asserted on a steady-state reconcile.
`stageset_update_deferred_total`	counter	`namespace`, `name`	Rollouts held by a closed update window.
`stageset_webhook_cert_renewal_failures_total`	counter	(none)	Failed self-signed webhook cert renewals.
`stageset_stage_ready`	gauge	`namespace`, `stageset`, `stage`	`1` when a stage is Ready, else `0` — for metric-based progressive delivery.

Alerts

The chart ships an opt-in PrometheusRule with a starter alert set, gated on metrics.prometheusRule.enabled (requires the Prometheus operator CRDs). It covers the custom stageset_* metrics plus controller-runtime signals:

Parameterizing a rollout

Mon, 01 Jan 0001 00:00:00 +0000

A rollout takes parameters at two distinct layers, which serve different purposes:

Render-time parameters (JaaS). Change what gets rendered. The Jsonnet computes its output from top-level arguments (tlas) and external variables (externalVariables). Different values produce a different ExternalArtifact.
Delivery-time parameters (StageSet postBuild). Inject values into already-rendered manifests, per stage, by string substitution — the same mechanism Flux’s kustomize-controller uses.

Use render-time parameters for structural logic; use delivery-time parameters to stamp environment-specific values onto a shared artifact.

PreviousRevisionUnavailable

Mon, 01 Jan 0001 00:00:00 +0000

Symptom

READY=False, REASON=PreviousRevisionUnavailable. The StageSet has spec.rollbackOnFailure set, a run failed, and the controller could not restore the last-good revisions.

Cause

rollbackOnFailure restores the previously-applied artifact revisions by re-fetching their recorded URLs and verifying their digests. That only works while the producer still retains those revisions. This reason means a revision the rollback needs is no longer fetchable — the producer garbage-collected it.

Rollback is best-effort by contract: it works exactly when producers retain. Common triggers:

Producer-aware sources

Mon, 01 Jan 0001 00:00:00 +0000

Stages and sources covers the two direct routes — an ExternalArtifact (the default sourceRef.kind) or a Flux GitRepository/OCIRepository/Bucket. This page covers the third: naming the thing that produces an artifact and letting the controller find it. This is useful when an operator publishes an ExternalArtifact from a custom resource (for example JaaS rendering Jsonnet).

Referencing a producer

Set kind (and apiVersion) to a producer resource, and the controller resolves it to the ExternalArtifact that producer publishes — the one whose spec.sourceRef back-references the producer (matched on group, kind, and name). For example, a JaaS JsonnetSnippet renders Jsonnet and publishes an ExternalArtifact; reference the snippet and the controller follows the link:

Production

Mon, 01 Jan 0001 00:00:00 +0000

High availability

The controller supports leader-elected HA. Enable leader election and run more than one replica; only the lease holder reconciles, while every replica answers admission webhook calls (admission must stay available even on non-leaders).

Leader election is toggled with --leader-elect. The binary defaults it to false, but the Helm chart enables it by default (controller.leaderElect: true), so a default install is already lease-guarded even at one replica.
The lease is named stageset-controller.stages.metio.wtf and lives in the controller’s namespace. It uses controller-runtime’s default timing (~15 s lease duration). The lease is not released eagerly on shutdown, so after a rolling update the new leader takes over when the old lease expires — budget a few seconds of reconcile pause on restart (admission and the gate endpoint are unaffected).
Scaling: when the chart’s replicas.max exceeds replicas.min it renders a HorizontalPodAutoscaler (CPU target 80%) and a PodDisruptionBudget (minAvailable: 1). At the default 1/1 it sets neither and leaves spec.replicas unmanaged.

The controller watches every namespace by default. Multi-tenancy is enforced per StageSet through impersonation (see below). You can additionally scope the controller to a namespace set with controller.watchNamespaces — one controller instance per tenant-group — and run it under cluster-admin for single-tenant clusters; both are covered in multi-cluster and tenancy.

Progressive delivery

Mon, 01 Jan 0001 00:00:00 +0000

StageSet integrates with both progressive-delivery controllers: Flagger and Argo Rollouts. The controller exposes a read-only gate endpoint and a readiness gauge so either one can hold a promotion until a StageSet stage is healthy; ready checks let a stage wait on a Rollout in return. Pick the section for your controller below — see also StageSet vs Argo Rollouts.

The gate contract

The gate endpoint backs the Flagger integration and the Argo Rollouts JSON-metric option.

Ready checks

Mon, 01 Jan 0001 00:00:00 +0000

Ready checks decide when a stage is healthy enough to let the next stage start. They are purely observational — the controller waits and reports, but takes no action (active steps are actions).

By default, with no readyChecks block, the controller waits for every object the stage applied to report ready via kstatus. readyChecks lets you narrow that to specific objects (checks), add custom health for resources kstatus doesn’t understand (exprs, CEL), bound the wait (timeout), or skip it entirely (disableWait). checks and exprs may be set together.

Reconcile latency high

Mon, 01 Jan 0001 00:00:00 +0000

Symptom

controller_runtime_reconcile_time_seconds p99 for controller="stageset" exceeds the configured threshold; the StageSetReconcileLatencyHigh alert fires (see operations for the alert set and its thresholds).

Cause

A single reconcile does a lot of work — resolve and fetch every stage’s artifact, kustomize-build, server-side apply, prune, verify readiness, and run actions — all impersonating the tenant ServiceAccount. Latency climbs when any of those is slow:

large artifacts or slow artifact servers,
many objects per stage (apply + prune scale with object count),
readiness waits and wait/http/job actions with long timeouts,
apiserver or tenant-authorization slowness.

Diagnosis

kubectl -n stageset-system logs deploy/stageset-controller --tail=200 | grep -i 'slow\|timeout\|took'

Break the latency down by stage count and artifact size; a single StageSet with many large stages dominates p99.

ResolveFailed

Mon, 01 Jan 0001 00:00:00 +0000

Symptom

READY=False, REASON=ResolveFailed. The Message describes why resolution failed.

Cause

A stage’s sourceRef could not be resolved to an ExternalArtifact for a spec/config or API reason (distinct from “not published yet”, which is SourceNotReady, and “no such object”, which is ArtifactNotFound). Common cases:

an ambiguous producer — more than one ExternalArtifact back-points at the same producer object, so the target is undefined;
a cross-namespace ref rejected by --no-cross-namespace-refs;
an API error reading the source or artifact (RBAC denial, the artifact CRD not installed).

When the failing sourceRef targets another namespace, the Message is deliberately scrubbed to cross-namespace <kind> %q is not reachable so tenants cannot fingerprint other namespaces — check that source CR’s status in its own namespace.

Rollback

Mon, 01 Jan 0001 00:00:00 +0000

When a run fails, the controller can restore the last successfully-applied artifact revisions instead of leaving you on a broken release. Rollback is opt-in and needs somewhere to keep prior revisions.

Enabling it

spec:
 rollbackOnFailure: true
 stages:
 - name: app
 sourceRef:
 name: my-app

On a failed run the controller restores each stage’s last-good artifact revision, best-effort, and emits a RolledBack event. The coordinates it restores from are recorded in status.lastAppliedSnapshot.

Secrets encryption (SOPS)

Mon, 01 Jan 0001 00:00:00 +0000

A stage’s source can carry SOPS-encrypted files — typically a Secret whose values are encrypted — and the controller decrypts them in memory, before building and applying the manifests. This mirrors Flux’s kustomize-controller decryption contract, so an existing SOPS-encrypted repository works unchanged.

Set spec.decryption and point it at a Secret holding the keys:

apiVersion: stages.metio.wtf/v1
kind: StageSet
metadata:
 name: payments
 namespace: payments
spec:
 serviceAccountName: payments-deployer
 decryption:
 provider: sops  # the only provider
 secretRef:
 name: sops-age  # a Secret in this namespace holding the age key
 stages:
 - name: app
 sourceRef:
 kind: GitRepository
 name: payments-config  # contains an encrypted secret.yaml

Walkthrough — age

age is the simplest key type and needs no external service. Take a Secret from plaintext to a GitOps-safe rollout in four steps.

SourceNotReady

Mon, 01 Jan 0001 00:00:00 +0000

Symptom

READY=False, REASON=SourceNotReady. Transient: the controller requeues and clears the condition once the source publishes.

Cause

A stage’s sourceRef resolved to an ExternalArtifact (directly, or via a producer’s RFC-0012 back-pointer such as a JaaS JsonnetSnippet), but that artifact’s status.conditions[Ready] is not yet True — its producer has not finished publishing a revision. The StageSet gates on Ready=True so it never builds against a half-written artifact.

Diagnosis

# Which artifact, and is it Ready?
kubectl get externalartifact -n <namespace>
kubectl describe externalartifact <name> -n <namespace>

# If the producer is a JsonnetSnippet (or other producer kind), check it:
kubectl describe jsonnetsnippet <name> -n <namespace>

Remediation

This usually clears on its own when the producer publishes. If it persists:

Stage sources — Git, OCI, Bucket

Mon, 01 Jan 0001 00:00:00 +0000

A stage resolves its sourceRef to a Flux artifact. You have two routes:

manifests in Git / OCI / Bucket ──────────────────────────► StageSet (direct)
manifests in Git / OCI / Bucket ──► a renderer (JaaS) ──► ExternalArtifact ──► StageSet

Use the direct route when the source already holds ready-to-apply manifests (the same thing Flux’s kustomize-controller consumes). Use the renderer route when you generate manifests first — e.g. evaluating Jsonnet with JaaS.

This page is the copy-pasteable recipe per source kind. For how sourceRef resolution works as a concept — and the path, prune, patches, and postBuild knobs that shape a stage — see stages and sources.

StageFailed

Mon, 01 Jan 0001 00:00:00 +0000

Symptom

READY=False, REASON=StageFailed. The Message names the stage and the operation that failed (fetch artifact, build, apply, verify, a pre/post action, or connect to target cluster). The run halts at that stage; later stages keep their previous revisions.

Cause

A stage failed during execution. By operation:

fetch artifact — the artifact URL was unreachable, or its bytes failed digest verification.
build — kustomize build or post-build substitution failed (a missing substituteFrom source, an invalid patch, a malformed manifest).
apply — the server-side apply was rejected: an immutable-field conflict, or an RBAC denial under the impersonated serviceAccountName.
verify — applied objects did not become Ready within the stage timeout (kstatus).
pre/post action — a patch/http/wait/job/delete/apply action failed or timed out.
connect to target cluster — a spec.kubeConfig Secret was missing, unparseable, or used the unsupported cloud-provider configMapRef.

Diagnosis

kubectl describe stageset <name> -n <namespace> # Message: which stage + operation
kubectl -n stageset-system logs deploy/stageset-controller --tail=200

# For apply/verify failures, inspect what the stage tried to apply:
kubectl get stageinventory -n <namespace> \
 -l stages.metio.wtf/stage-set=<name>,stages.metio.wtf/stage=<stage>

Remediation

Match the operation in the Message:

StageInventory

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: stages.metio.wtf/v1
kind: StageInventory

A StageInventory records the set of objects a single stage has applied, so the controller can prune precisely and tear stages down in reverse order. You do not author these — the controller creates, updates, and deletes them. They are documented here so you can read them when debugging and back them up.

One stage may be backed by several StageInventory shards once it exceeds --inventory-shard-cap entries (default 5000). Shard 0 doubles as the ApplySet (KEP-3659) parent object for the stage.

Stages and sources

Mon, 01 Jan 0001 00:00:00 +0000

A StageSet is an ordered list of stages. Each stage resolves a Flux source — a GitRepository, OCIRepository, Bucket, or an ExternalArtifact (the default) — applies its manifests, waits for them to become healthy, and only then lets the next stage start.

One stage

The minimum is one stage pointing at one artifact in the same namespace:

apiVersion: stages.metio.wtf/v1
kind: StageSet
metadata:
 name: my-app
 namespace: default
spec:
 stages:
 - name: app
 sourceRef:
 name: my-app  # an ExternalArtifact

sourceRef.kind defaults to ExternalArtifact, so the common case is a single line. The controller fetches the artifact, applies every manifest in it, and marks the stage Ready once the applied objects report healthy.

StageSet

Mon, 01 Jan 0001 00:00:00 +0000

apiVersion: stages.metio.wtf/v1
kind: StageSet

A StageSet is a namespaced Kubernetes resource describing an ordered set of stages. Only spec.stages is required; everything else refines scheduling, security, gating, versioning, and rollback. Every field below is shown in YAML at least once.

The smallest valid StageSet:

apiVersion: stages.metio.wtf/v1
kind: StageSet
metadata:
 name: my-app
 namespace: default
spec:
 stages:
 - name: app
 sourceRef:
 name: my-app

Scheduling

spec:
 interval: 5m # optional: reconcile cadence (default: --default-interval)
 retryInterval: 1m # cadence after a failed run (default: interval)
 driftDetectionInterval: 2m  # faster drift correction than interval (optional)
 timeout: 5m  # default per-stage timeout (optional)
 suspend: false # pause reconciliation without deleting (default false)

interval (optional) — steady-state reconcile cadence; each reconcile re-resolves sources, re-asserts desired state (correcting drift), and prunes. When omitted, the controller’s --default-interval is used (the chart’s controller.defaultInterval, default 10m), so most StageSets can leave it out.
retryInterval — retry cadence after a failure; falls back to interval.
driftDetectionInterval — a shorter cadence dedicated to healing out-of-band drift when you need it tighter than interval.
timeout — how long any one stage may take before it fails; override per stage with stages[].timeout.
suspend — short-circuits to Ready=False / Suspended, leaving applied state running. Use stagesetctl reconcile --force to run once while suspended. See the Suspended runbook.

Ordering between StageSets

dependsOn gates this StageSet on others being Ready at their observed generation — cross-release ordering. (Ordering within a StageSet is the order of stages.)

StageSet vs Argo Rollouts

Mon, 01 Jan 0001 00:00:00 +0000

Argo Rollouts and StageSet are easy to mention in the same breath because both roll things out gradually, but they operate at different layers and are complementary rather than competing.

What Argo Rollouts does

Argo Rollouts replaces a Deployment with a Rollout that shifts traffic to a new version progressively — canary or blue-green — pausing at weighted steps and promoting based on metric analysis (Prometheus queries, web/Job providers). Its unit of work is a single workload’s version transition and the traffic in front of it.

StageSet vs Flux kustomize-controller

Mon, 01 Jan 0001 00:00:00 +0000

This is the closest comparison — StageSet is built for Flux and borrows its conventions. Flux’s kustomize-controller (and helm-controller) reconcile a source into the cluster continuously, exactly like StageSet. The difference is granularity.

What kustomize-controller gives you

Continuous reconciliation of a Kustomization from a Flux source, with pruning, health checks, drift correction, and dependsOn ordering between Kustomizations.
Impersonation via serviceAccountName, postBuild substitution, patches — the same surface StageSet deliberately mirrors.

Where StageSet differs

Ordering within a release. kustomize-controller applies one Kustomization as a unit; ordering exists only between Kustomizations via dependsOn. To sequence three steps you create three Kustomizations and wire their dependencies. StageSet expresses that as one resource with ordered stages — and the controller waits for each stage’s health before the next.
Typed actions between steps. Migrations, HTTP gates, waits, and transient applies are first-class actions; in plain Flux you’d model these as extra Kustomizations and Jobs.
Release-level features. Update windows, versioned migrations, and rollback operate across the whole staged release.
Source-native. A stage consumes a GitRepository/OCIRepository/Bucket directly (just like kustomize-controller), or an ExternalArtifact (RFC-0012), or a producer resolved to its artifact — which is how it also pairs with renderers like JaaS.
SOPS parity. Encrypted Secrets in a source decrypt the same way, via spec.decryption (age, PGP, or cloud KMS), so a SOPS-using repo ports across unchanged.

Using them together

StageSet sits alongside the other Flux controllers and reuses Flux’s source layer, notifications (Alert/Provider targeting kind: StageSet), and reconcile annotations. Use kustomize-controller for ordinary one-shot reconciliation and reach for StageSet when a release needs ordered, gated stages.

StageSet vs Helm

Mon, 01 Jan 0001 00:00:00 +0000

Helm is two things: a templating engine (charts) and an imperative release tool (helm upgrade). StageSet is neither — it’s a declarative delivery controller. The overlap is ordering: Helm’s hooks and hook weights give you some sequencing inside a single chart’s install/upgrade.

What Helm gives you

Templated, parameterized manifests (charts and values).
Install/upgrade ordering via helm.sh/hook (pre-install, post-upgrade, …) and hook-weight.
A release history you can roll back to with helm rollback.

Where StageSet differs

Continuous reconciliation. helm upgrade is a point-in-time, imperative action; nothing re-asserts the state afterward. StageSet reconciles on an interval, corrects drift, and prunes — it’s GitOps, not a one-shot.
Ordering across artifacts, not just within one chart. Helm hooks order resources inside a release. StageSet orders whole stages, each its own artifact, with readiness gating between them.
Typed gates between steps. Hooks run Jobs; StageSet stages can run Jobs, HTTP gates, waits, patches, deletes, and transient applies, as pre/post/onFailure actions.
Identity. A StageSet applies under an impersonated, per-tenant ServiceAccount; helm upgrade runs as whoever ran it.

Using them together

Render a chart to manifests (e.g. via a producer that publishes an ExternalArtifact) and deliver it with StageSet. The controller understands helm.sh/hook resources: applyHelmHookResources (default true) applies them as ordinary objects, so a Helm-style chart’s hook resources still get created — now under StageSet’s ordering and gating instead of Helm’s.

StageSet vs jsonnet-controller

Mon, 01 Jan 0001 00:00:00 +0000

jsonnet-controller (pelotech) is a Flux controller that evaluates Jsonnet (kubecfg- and Tanka-style) and applies the result to the cluster. Its Konfiguration resource (jsonnet.io/v1beta1) is, in effect, kustomize-controller for Jsonnet: point it at a GitRepository (or an HTTP(S) URL), and it builds the Jsonnet and reconciles the manifests — with pruning, health/revision tracking, TLA string/code variables, and dependsOn ordering between Konfigurations.

The two projects sit at different layers, which is the whole comparison.

StageSet vs Kustomize

Mon, 01 Jan 0001 00:00:00 +0000

Kustomize (the kustomize CLI / kubectl kustomize) is a manifest builder: it composes bases and overlays, applies patches, and emits YAML. It does not apply anything, and it has no notion of ordering, readiness, or reconciliation — that’s kubectl apply’s job, and kubectl applies everything at once.

What Kustomize gives you

Overlay composition, strategic-merge and JSON6902 patches, variable replacement, generators.
A pure transformation: in goes a kustomization, out come manifests.

Where StageSet differs

It delivers, not just builds. Kustomize stops at YAML. StageSet applies it, waits for health, prunes what’s gone, and keeps doing so.
Ordering and gates. kubectl apply -k has no stages and no gates. StageSet sequences stages and runs actions between them.
Continuous reconciliation and drift correction, versus a one-shot apply.

Using them together

StageSet includes the parts of Kustomize you reach for at delivery time: a stage has path, patches, and postBuild substitution (stages and sources). So you can keep authoring with Kustomize overlays and let a stage apply the right overlay, patched and substituted — then add the ordering, gating, and reconciliation Kustomize alone doesn’t offer.

StageSet vs Tanka and kubecfg

Mon, 01 Jan 0001 00:00:00 +0000

Tanka and kubecfg are Jsonnet-based config tools: you express your resources in Jsonnet, the tool renders them, diffs against the cluster, and applies. They generate configuration and run a CLI-driven apply, but they are imperative tools you run, not controllers that reconcile.

What Tanka / kubecfg give you

Jsonnet-powered, DRY manifest generation (libraries, abstractions, environments).
A diff/apply workflow with dependency-aware ordering of a single apply.

Where StageSet differs

Reconciliation, not invocation. Tanka/kubecfg apply when you run them. StageSet runs in-cluster and continuously reconciles, corrects drift, and prunes.
Staged, gated delivery. They apply a rendered set (in dependency order); they don’t model multi-stage rollouts with readiness gates, update windows, or versioned migrations between stages.
GitOps identity and tenancy. StageSet applies under an impersonated tenant ServiceAccount inside the cluster; Tanka/kubecfg use your local credentials.

Using them together

The Jsonnet generation that Tanka and kubecfg do so well has a GitOps-native equivalent in two related projects:

stagesetctl build

Mon, 01 Jan 0001 00:00:00 +0000

Runs the same resolve → fetch → build pipeline the controller uses and writes the result — a multi-document YAML stream — to stdout. This is what would be applied, before it is applied. To preview the change against live cluster state instead, use diff.

stagesetctl build NAME [flags]

Flag	Default	Description
`--stage`	(all)	Render only the named stage(s); repeatable.
`--source-dir`	(none)	Use a local artifact tree as `[STAGE=]PATH` instead of fetching from the cluster; repeatable.
`--show-secrets`	`false`	Reveal Secret values instead of masking them.
`--as-tenant`	`false`	Render impersonating the StageSet’s `spec.serviceAccountName` (see multi-cluster and tenancy).

Secret values are masked by default, so the output is safe to paste into a review. build writes YAML unconditionally — there is no output-format flag.

stagesetctl diff

Mon, 01 Jan 0001 00:00:00 +0000

By default diff performs a server-side dry-run apply and exits 1 when there are changes, so it works as a CI gate. It shows, per object, what a reconcile would create, configure, or delete, plus the actions a rollout would run. To see the full rendered manifests without comparing against the cluster, use build.

stagesetctl diff NAME [flags]

Flag	Default	Description
`--stage`	(all)	Diff only the named stage(s); repeatable.
`--source-dir`	(none)	Use a local artifact tree as `[STAGE=]PATH`; repeatable. Skips the cluster fetch.
`--server-side`	`true`	Server-side dry-run apply diff (needs update/patch RBAC). `false` renders client-side against live objects.
`--as-tenant`	`false`	Render and dry-run impersonating `spec.serviceAccountName` (see multi-cluster and tenancy).
`--show-secrets`	`false`	Reveal Secret values instead of masking.
`--show-unchanged`	`false`	Include objects with no change.
`--prune`	`true`	Show resources that would be deleted (fell out of inventory).
`--color`	`auto`	Colorize output: `auto`, `always`, or `never`.
`--exit-code`	`true`	Exit `1` when changes are found. `false` always exits `0` on a clean run.

Example

stagesetctl diff payments

--- live
+++ merged
@@ Deployment payments/web @@
 spec:
- replicas: 3
+ replicas: 6

- ConfigMap payments/old-feature-flags (pruned: fell out of inventory)

Actions to run:
 application:
 pre db-migrate job ledger-migrations
 post smoke-test http https://payments.internal/healthz

Objects that left the stage’s inventory show as deletions (pruned: …); pass --prune=false to hide them. The trailing Actions to run block lists the pre/post/onFailure actions a real reconcile would execute — diff never runs them, it only reports them.

stagesetctl get

Mon, 01 Jan 0001 00:00:00 +0000

With no NAME, lists StageSets in the current namespace. With a NAME, prints that StageSet’s detail (Ready reason, per-stage phase, revisions, version) — a readable view of StageSet.status.

stagesetctl get [NAME] [flags]

Flag	Default	Description
`-A`, `--all-namespaces`	`false`	List StageSets across all namespaces.
`-o`, `--output`	(table)	Output format: empty for the human table, or `yaml` / `json`.

Listing

stagesetctl get -A

NAMESPACE NAME READY REASON STAGES VERSION PENDING
payments payments True Succeeded 2/2 2.1.0 -
platform platform True Succeeded 3/3 - -
staging web False StageFailed 1/2 - -

STAGES is ready/total; PENDING shows held until <time> when an update window is holding a rollout. A False READY maps to a runbook by its REASON.

stagesetctl reconcile

Mon, 01 Jan 0001 00:00:00 +0000

Stamps the reconcile.fluxcd.io/requestedAt annotation to trigger a reconcile now, optionally waiting for the controller to report it handled.

stagesetctl reconcile NAME [flags]

Flag	Default	Description
`--stage`	(all)	Force only this stage to re-run its actions (single-stage reconcile).
`--with-source`	`false`	Also re-request the stage sources before reconciling.
`--update-now`	`false`	Apply a window-held rollout immediately, bypassing update windows.
`--force`	`false`	Proceed even when the StageSet is suspended.
`--wait`	`false`	Block until the controller reports the request handled.
`--timeout`	`5m`	How long to wait with `--wait`.

Example

stagesetctl reconcile payments -n payments

Reconcile requested for StageSet payments (token 2026-06-15T09:30:00Z)

Force just one stage to re-run its actions:

Stalled

Mon, 01 Jan 0001 00:00:00 +0000

Symptom

READY=False, REASON=Stalled. Terminal: the controller does not requeue until the spec changes.

Cause

A condition that retrying cannot clear. Currently this is a spec.dependsOn cycle — two or more StageSets depend on each other (directly or transitively), so none can ever become Ready first. The cycle is detected by a breadth-first walk over the dependsOn graph. A dependency that is merely not Ready yet (no cycle) reports DependencyNotReady instead.

Diagnosis

kubectl describe stageset <name> -n <namespace> # Message states "spec.dependsOn forms a cycle"
# Trace the edges:
kubectl get stageset -n <namespace> \
 -o custom-columns=NAME:.metadata.name,DEPENDSON:.spec.dependsOn[*].name

Follow the dependsOn names until you find the loop (A → B → A, or longer).

Succeeded

Mon, 01 Jan 0001 00:00:00 +0000

Symptom

READY=True, REASON=Succeeded. The Message names the applied revisions.

Cause

This is the healthy steady state: every stage’s artifact resolved, built, applied, and passed its readiness checks, and status.lastAppliedRevisions matches status.lastAttemptedRevisions.

Remediation

Nothing to remediate.

The controller keeps reconciling at spec.interval; a re-render upstream (a new ExternalArtifact revision) re-applies automatically and the condition stays Succeeded once the new revision converges.
status.stages[] reports per-stage appliedRevision and inventory entry counts to confirm what each stage owns.

Suspended

Mon, 01 Jan 0001 00:00:00 +0000

Symptom

READY=False, REASON=Suspended.

Cause

spec.suspend: true is set, so the controller short-circuits before any resolution, build, or apply. This is an intentional operator action, not a failure — applied objects are left exactly as they were at the last successful run.

Remediation

Resume by clearing the flag:

kubectl patch stageset <name> -n <namespace> --type=merge -p '{"spec":{"suspend":false}}'

The next reconcile picks up from the current artifact revisions.

Update windows

Mon, 01 Jan 0001 00:00:00 +0000

Update windows gate when new artifact revisions roll out, without pausing reconciliation. Drift correction keeps running; only the rollout of a new revision is held until a window allows it.

Deny a recurring window

Freeze rollouts during business hours:

spec:
 stages:
 - name: app
 sourceRef:
 name: my-app
 updateWindows:
 - type: Deny
 schedule: "0 9 * * MON-FRI" # 5-field cron: start of the window
 duration: 8h
 timeZone: Europe/Berlin

A new revision that arrives inside the window is held; status.pendingUpdate records what is waiting and nextWindowOpens when it will ship. The controller emits an UpdateDeferred event and increments stageset_update_deferred_total.

UpdateDeferred

Mon, 01 Jan 0001 00:00:00 +0000

Symptom

READY=False, REASON=UpdateDeferred (initial deploy held), or READY=True with a message noting a deferral and a populated status.pendingUpdate (an already-deployed StageSet with a held update).

Cause

This is not a failure — it is time-based delivery working as configured. A new revision (or the first deploy) is being held because the StageSet’s spec.updateWindows do not currently permit a rollout: either a Deny window is active, or Allow windows are declared and none is active right now. With spec.windowScope: All, even drift correction is paused while a window is closed.

Versioned migrations

Mon, 01 Jan 0001 00:00:00 +0000

Some changes only need to happen once, when you cross a release boundary — a one-time data backfill on the way to 2.0, a schema conversion between 1.x and 2.x. Versioned migrations run a ladder of actions exactly when the deployed version steps over the boundary, and never again.

Versioning is off until you set spec.version.

Declaring the version

The controller needs to know what version is currently being deployed. There are three ways to declare it; pick by where the version lives.

Webhook cert renewal failing

Mon, 01 Jan 0001 00:00:00 +0000

Symptom

stageset_webhook_cert_renewal_failures_total is increasing; the StageSetWebhookCertRenewalFailing alert fires (see operations for the alert set and its thresholds). The current certificate keeps working until its natural expiry — that expiry is the deadline, after which cluster-wide StageSet admission breaks.

Cause

Only applies in --webhook-cert-mode=self-signed. The in-pod renewer regenerates the serving cert every validity/3 and patches the ValidatingWebhookConfiguration’s caBundle. It fails when:

the controller lost update (or get) on the named ValidatingWebhookConfiguration (--webhook-validating-config-name),
the VWC was renamed and the flag/resourceNames weren’t updated,
the cert directory (--webhook-cert-dir) became read-only.

In cert-manager mode this metric is irrelevant — cert-manager owns renewal.

Workqueue saturation

Mon, 01 Jan 0001 00:00:00 +0000

Symptom

workqueue_depth{controller="stageset"} stays high; StageSets reconcile slowly or lag behind their spec.interval. The StageSetControllerWorkqueueDepthHigh alert fires (see operations for the alert set and its thresholds).

Cause

The controller is enqueuing reconcile requests faster than it completes them. Common causes:

apiserver slowness — applies, dry-runs, and status writes all block on the apiserver (or the impersonated tenant’s authorization).
slow sources — a stage waiting on a large artifact fetch or a source that is slow to become Ready holds a worker.
a stuck stage — an action with a long timeout (a wait/http/job that never completes) pins a worker for the whole timeout.
too few workers for the StageSet count — many StageSets reconciling on short intervals.

Diagnosis

# which StageSets are churning?
kubectl get stagesets -A --sort-by=.status.observedGeneration
# controller logs for slow operations / retries
kubectl -n stageset-system logs deploy/stageset-controller --tail=200

Correlate with controller_runtime_reconcile_time_seconds (see reconcile latency) and apiserver latency.