Actions are typed steps the controller runs around a stage’s apply. They turn an ordered apply into an orchestrated rollout — run a migration before the app, gate the stage on an external check, clean up on failure.

A stage has three action hooks:

• pre — run before the manifests are built and applied. A failure aborts the stage with nothing applied.
• post — run after the apply is verified. The stage is Ready only if these all succeed.
• onFailure — best-effort steps run on any failure from the apply onward.

Each action has a name, optional timeout and retries, and exactly one operation type (patch, http, wait, job, delete, or apply) — enforced by the validating admission webhook. Actions within a hook run in list order.

spec:
  stages:
    - name: application
      sourceRef:
        name: my-app
      actions:
        pre:
          - name: db-migrate
            timeout: 10m
            job:
              sourceRef:
                name: my-app-migrations    # render & await Jobs from this artifact
        post:
          - name: smoke-test
            retries: 3
            http:
              url: https://my-app.internal/healthz
              expectedStatus: [200]
        onFailure:
          - name: page-oncall
            http:
              url: https://alerts.internal/stageset-failed
              method: POST

The six action types

job

Render and await Kubernetes Jobs from an artifact. The classic use is a database migration that must complete before the app is applied.

- name: db-migrate
  job:
    sourceRef:
      name: my-app-migrations
    path: ./jobs

http

Call an HTTP endpoint and gate on the response — an approval webhook, a smoke test, an external readiness probe. Hosts must be permitted by the controller’s --allowed-action-hosts; loopback and link-local are always denied. method defaults to POST; expectedStatus defaults to any 2xx. The body and headers can be read from a Secret so tokens never sit in the spec:

- name: smoke-test
  http:
    url: https://my-app.internal/healthz
    method: GET
    expectedStatus: [200]
    headersFrom:
      - name: gate-token       # Secret name
        key: authorization     # the key names the header; its value is the value
    bodyFrom:
      name: gate-payload       # Secret name
      key: body                # this key's value becomes the request body

wait

Block for a fixed duration, or until a CEL expression holds against a target object.

- name: settle
  wait:
    duration: 30s
- name: until-available
  wait:
    target:
      apiVersion: apps/v1
      kind: Deployment
      name: web
    expr: "status.availableReplicas >= 3"
    timeout: 5m

patch

Patch an existing object — flip a feature flag, scale something, annotate. type is merge (default) for a strategic-merge patch, or json6902 for a JSON Patch:

- name: enable-traffic
  patch:
    target:
      apiVersion: v1
      kind: Service
      name: web
    type: merge               # default; or json6902
    patch: |
      { "spec": { "selector": { "release": "green" } } }

delete

Remove an existing object; a missing object counts as success.

- name: drop-old-job
  delete:
    target:
      apiVersion: batch/v1
      kind: Job
      name: legacy-migration

apply

Apply transient, rollout-scoped manifests that are not inventory-tracked and are never pruned — a maintenance page, a one-shot canary, a temporary config. With wait: true the action blocks until the applied objects report Ready (kstatus), bounded by the action timeout, so a following patch can repoint traffic only once the resource is serving.

Because the applied objects are never pruned by the inventory diff, stand a resource up only for the duration of a rollout by pairing an apply in pre with a matching delete in post, and guard a mid-run crash with an onFailure delete:

actions:
  pre:
    - name: stand-up-maintenance-page
      apply:
        sourceRef:
          name: maintenance-page    # an ExternalArtifact holding a Pod + Service
        wait: true                  # block until it is serving
  post:
    - name: tear-down-maintenance-page
      delete:
        target:
          apiVersion: v1
          kind: Pod
          name: maintenance-page
  onFailure:
    - name: tear-down-maintenance-page-on-failure
      delete:
        target:
          apiVersion: v1
          kind: Pod
          name: maintenance-page

The action ledger gates each step per pinned revision, so a retry or controller restart never re-applies or re-deletes the resource for the same snapshot.

To run a job action only when the deployed version crosses a release boundary, see versioned migrations.

Symptom

READY=False, REASON=ArtifactNotFound. Transient: the controller requeues in case the artifact appears.

Cause

A stage’s sourceRef resolves to no ExternalArtifact. Either:

• a direct sourceRef (kind: ExternalArtifact, the default) names an object that does not exist in the target namespace; or
• a producer sourceRef (e.g. kind: JsonnetSnippet) exists, but no ExternalArtifact carries a spec.sourceRef back-pointer to it yet — the producer has not created its artifact object.

Diagnosis

kubectl describe stageset <name> -n <namespace>     # Message names the missing ref
kubectl get externalartifact -n <namespace>

For a producer ref, confirm the producer object exists and that it is configured to publish an ExternalArtifact (not only serve over HTTP):

kubectl get <producer-kind> <name> -n <namespace> -o yaml

Remediation

• Fix a typo in sourceRef.name / sourceRef.namespace.
• For a direct ref, create (or wait for) the named ExternalArtifact.
• For a producer ref, ensure the producer actually publishes an artifact and that it lands in the same namespace as the StageSet (cross-namespace producer refs are gated by --no-cross-namespace-refs).

If the artifact exists but is not yet published, the reason is SourceNotReady; a spec/API resolution failure is ResolveFailed. See stages and sources.

The controller is a standard Go module. With a Go toolchain installed:

go build ./...
go test -race -cover ./...

Test layers

• Unit tests sit next to the code across internal/... and api/v1/. Several are drift gates — e.g. conditions_test.go asserts every Ready Reason has a matching runbook page under docs/content/runbooks/.
• envtest-backed tests (envtest_*_test.go) boot a real kube-apiserver + etcd via controller-runtime’s envtest. They t.Skip unless KUBEBUILDER_ASSETS points at an asset bundle — install it with setup-envtest.
• Fuzz tests (FuzzXxx) harden the parsing-heavy paths; their seed corpus runs as ordinary unit tests, and -fuzz fuzzes for real.
• Kind smoke scenarios under hack/smoke/ run the controller end to end against a real kind cluster.

Static analysis

A pull request must be clean under each of these — run them locally before pushing:

go vet ./...
go run honnef.co/go/tools/cmd/staticcheck@latest ./...
go run github.com/securego/gosec/v2/cmd/gosec@latest ./...
go run golang.org/x/vuln/cmd/govulncheck@latest ./...
go run mvdan.cc/gofumpt@latest -l .          # empty output == formatted
go run github.com/fe3dback/arch-go@latest    # architecture rules (arch-go.yml)

Containerized dev shell

The toolchain — including the pinned setup-envtest assets — is also packaged in a container via ilo, so you can build and test without installing anything on the host:

ilo bash -c 'go test -race -cover ./...'

Continuous integration

Every pull request runs verify.yml, which fans out into one job per concern so a failure points straight at the cause:

• test — go build then the full go test suite.
• lint-go — go vet, staticcheck, gosec, and a gofumpt formatting check.
• vulnerabilities — govulncheck (a reachable advisory is a hard gate).
• architecture — arch-go against arch-go.yml.
• reuse — SPDX/REUSE compliance on every file.
• text linters — yamllint, actionlint, markdownlint, typos.
• container-image — a buildx image build plus a Trivy scan.

A single all-green job depends on every other job and is the only required check, so new jobs are covered automatically. A separate kind-smoke.yml runs the operator end to end against a real kind cluster, and fuzz.yml exercises the fuzz targets.

Releases

Releases are calendar-based and fully automated — there is no semver tag to bump by hand. release.yml runs on a Monday cron (and on manual dispatch), and the version is the run date (date +'%Y.%-m.%-d', e.g. 2026.6.15). A prepare job counts commits since the last release; an empty week publishes nothing.

The pipeline is hand-rolled — no goreleaser, no GPG:

• Binaries are cross-compiled with go build (CGO_ENABLED=0, -trimpath, -ldflags) and archived per platform.
• A multi-arch image is pushed to ghcr.io/metio/stageset-controller and signed with cosign keyless (Fulcio OIDC) by digest.
• The GitHub Release attaches the archives, a SHA256SUMS file, and its cosign signature; identity is proven by the workflow’s OIDC certificate, so there is no key to distribute.

The Helm chart lives in the metio/helm-charts repository and vendors this repo’s CRDs at each release; its appVersion tracks the binary’s releases.

Conflict policies decide what happens when an apply hits an immutable-field conflict — a changed clusterIP, a Job pod template, a StorageClass field that can’t be updated in place. By default the controller fails the stage and reports it, so nothing destructive happens by surprise. A policy opts specific resources into automatic resolution.

The three actions

• Fail — stop and report (the default; safest).
• Recreate — delete and re-create the object to get past an immutable-field change.
• KeepExisting — leave the live object as-is and move on.

A default for the whole stage

spec:
  stages:
    - name: app
      sourceRef:
        name: my-app
      conflictPolicy:
        default: Fail            # explicit; the safe default

The force: true shorthand on a stage is equivalent to conflictPolicy.default: Recreate.

Per-resource rules

Rules recreate exactly the resources that need it while everything else stays Fail. A rule’s target is a partial selector — any field you omit matches everything. Rules are evaluated in list order; the first rule whose target matches wins, and an object matching no rule falls back to default.

      conflictPolicy:
        default: Fail
        rules:
          # a Job's pod template is immutable — recreate it on change
          - target:
              apiVersion: batch/v1
              kind: Job
            action: Recreate
          # never fight an HPA over replica counts
          - target:
              kind: Deployment
              name: web
            action: KeepExisting

Recreating storage

Recreating a PersistentVolumeClaim or PersistentVolume destroys data, so a Recreate rule targeting one is refused unless you explicitly accept the loss:

        rules:
          - target:
              kind: PersistentVolumeClaim
              name: scratch
            action: Recreate
            allowDataLoss: true     # required for PVC/PV Recreate, refused otherwise

Without allowDataLoss: true, a Recreate rule targeting a PVC/PV is rejected — a guardrail against accidentally wiping a volume.

Symptom

A stageset-controller pod is NotReady; the StageSetControllerPodDown alert fires. While no replica is Ready, StageSets are not reconciled and the Kubernetes admission webhook may reject StageSet writes (failurePolicy: Fail).

Cause

• a crash-looping container (bad config flag, missing RBAC, panic),
• the node draining or out of resources,
• a failing readiness probe (/readyz on --health-probe-bind-address),
• the leader-election lease unobtainable.

Diagnosis

kubectl -n stageset-system get pods -l app.kubernetes.io/name=stageset-controller
kubectl -n stageset-system describe pod <pod>
kubectl -n stageset-system logs <pod> --previous --tail=200

Look for flag-parse errors at startup, RBAC Forbidden on the controller’s own ServiceAccount, or OOMKills.

Remediation

• Fix the surfaced cause (correct the flag/values, grant the missing controller RBAC, raise resource limits).
• Run more than one replica with leader election so a single pod failure doesn’t stop reconciliation — see production.
• If admission is blocking writes during the outage and you must unblock urgently, scope or relax the webhook failurePolicy, then restore it once the controller is healthy.

See operations for the full alert set and its thresholds.

Symptom

READY=False, REASON=DependencyNotReady. Transient: the controller requeues at spec.retryInterval (or spec.interval).

Cause

A StageSet listed in spec.dependsOn is not Ready at its observed generation, so this StageSet holds before doing any work. Semantics match kustomize-controller: a dependency is satisfied only when its Ready=True and its status.observedGeneration equals its current generation (so a freshly-edited dependency mid-reconcile does not count as ready).

Diagnosis

kubectl describe stageset <name> -n <namespace>            # Message names the dependency
kubectl get stageset <dependency> -n <namespace>           # is it Ready?
kubectl describe stageset <dependency> -n <namespace>      # why not?

Remediation

Resolve the dependency’s own Ready condition first (follow its runbook). Once it reports Ready=True at its current generation, this StageSet proceeds on the next reconcile. If the dependency is intentionally suspended, this StageSet waits indefinitely by design — remove the dependsOn entry or resume the dependency.

A dependsOn cycle is reported as Stalled, not this reason.

Symptom

READY=False, REASON=DowngradeRequiresMigration. Terminal: the run does not requeue until the desired version is at or above status.version.

Cause

The desired version (spec.version) is lower than the version the controller last recorded as deployed (status.version). Downgrades are refused by default: migrations are forward-only action ladders, and replaying upgrade migrations in reverse is how data gets destroyed. The controller does not silently run a downgrade.

Diagnosis

kubectl describe stageset <name> -n <namespace>
kubectl get stageset <name> -n <namespace> -o jsonpath='{.status.version}'   # deployed
# desired: read spec.version.value, or the version file the artifact carries

Remediation

Pick the intended direction:

• You did not mean to downgrade (e.g. a source revert pulled an older version file): roll the source forward again so the desired version is >= the deployed version. The StageSet converges normally.
• You genuinely need to go back: a downgrade is an operational decision with potential data loss. Perform it deliberately — restore from backup or apply an explicit down-migration out of band — then set status.version to match. There is no automatic reverse-migration path by design.

This tutorial follows a complete delivery: write Kubernetes manifests in Jsonnet and publish the source through Flux; JaaS renders it into a Flux ExternalArtifact, and a StageSet rolls it out with a readiness gate.

The chain is:

Jsonnet in Git/OCI/Bucket  →  JaaS (JsonnetSnippet)  →  ExternalArtifact  →  StageSet

This tutorial renders Jsonnet, so it goes through JaaS: JaaS turns the Jsonnet into an ExternalArtifact the stage consumes. (If your manifests were already plain YAML, a stage could read a GitRepository/OCIRepository/Bucket directly — see Stage sources. The renderer is here because the input is Jsonnet, not because StageSet can’t read Git.)

Prerequisites

• Flux installed (with the ExternalArtifact API — Flux ≥ v2.7.0).

• JaaS installed in operator mode.

• StageSet installed (see Installation).

• An apps namespace, and a web-deployer ServiceAccount in it whose RBAC can apply the workload (the StageSet impersonates it):

  kubectl create namespace apps
  kubectl -n apps create serviceaccount web-deployer
  # bind web-deployer to a Role/ClusterRole that can manage Deployments and
  # Services in the apps namespace — see /usage/multi-cluster/ for the tenancy model
  

1. Write the manifests in Jsonnet

A small web app, parameterized as a Jsonnet top-level function so the same source renders for any environment. Commit this as jsonnet/main.jsonnet in a Git repo:

// jsonnet/main.jsonnet
function(name='web', image='registry.internal/web:latest', replicas='2') {
  apiVersion: 'v1',
  kind: 'List',
  items: [
    {
      apiVersion: 'apps/v1',
      kind: 'Deployment',
      metadata: { name: name },
      spec: {
        replicas: std.parseInt(replicas),
        selector: { matchLabels: { app: name } },
        template: {
          metadata: { labels: { app: name } },
          spec: { containers: [{ name: name, image: image }] },
        },
      },
    },
    {
      apiVersion: 'v1',
      kind: 'Service',
      metadata: { name: name },
      spec: { selector: { app: name }, ports: [{ port: 80, targetPort: 8080 }] },
    },
  ],
}

Rendering a kind: List keeps several resources in one document — both the kustomize build the controller runs and kubectl flatten it transparently.

2. Publish the source through Flux

Point a Flux GitRepository at the repo so the cluster has the Jsonnet:

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: web-manifests
  namespace: apps
spec:
  interval: 1m
  url: https://github.com/acme/web-manifests
  ref:
    branch: main

Apply it and wait for the source to sync:

kubectl apply -f gitrepository.yaml
kubectl -n apps wait --for=condition=Ready gitrepository/web-manifests

3. Render with JaaS

A JsonnetSnippet reads the Jsonnet from that source, passes the parameters as top-level arguments, and publishes the rendered result as an ExternalArtifact:

apiVersion: jaas.metio.wtf/v1
kind: JsonnetSnippet
metadata:
  name: web
  namespace: apps
spec:
  sourceRef:
    kind: GitRepository
    name: web-manifests
    path: ./jsonnet
  entryFile: main.jsonnet
  tlas:                            # top-level args → the function() parameters
    name: ["web"]
    image: ["registry.internal/web:2.1.0"]
    replicas: ["3"]

Apply it; JaaS then publishes an ExternalArtifact named web in the apps namespace. Confirm it went Ready:

kubectl apply -f jsonnetsnippet.yaml
kubectl -n apps get externalartifact web

4. Roll it out with StageSet

Reference the JsonnetSnippet as the stage source — StageSet resolves the producer to its ExternalArtifact — and gate the stage on the Deployment becoming available:

apiVersion: stages.metio.wtf/v1
kind: StageSet
metadata:
  name: web
  namespace: apps
spec:
  serviceAccountName: web-deployer      # applies are impersonated as this SA
  stages:
    - name: web
      sourceRef:
        apiVersion: jaas.metio.wtf/v1
        kind: JsonnetSnippet
        name: web
      readyChecks:
        checks:
          - apiVersion: apps/v1
            kind: Deployment
            name: web

Apply it, preview the change before it lands, then watch it roll out:

kubectl apply -f stageset.yaml
stagesetctl diff web -n apps          # preview against live cluster state
stagesetctl get  web -n apps          # per-stage progress

5. Ship a change

Edit jsonnet/main.jsonnet (or bump the image TLA on the snippet) and commit. Flux pulls the new commit, JaaS re-renders and republishes the ExternalArtifact, and StageSet — watching the producer — reconciles the new revision through the same gate. No StageSet edit required.

No labels or annotations needed

You do not annotate or label anything to make this chain fire. The linkage is the sourceRef itself: the controller watches the source kinds (ExternalArtifact, GitRepository, OCIRepository, Bucket, and producers like JsonnetSnippet) and, when one changes, maps it back to every StageSet whose sourceRef points at it — then reconciles those. JaaS works the same way for a snippet’s own sourceRef and library references. Discovery is automatic; you only declare the references.

Versioning the rollout

To gate one-time migrations on a release boundary, declare the version. The simplest is to pin it on the StageSet, bumped alongside the image:

spec:
  version:
    value: "2.1.0"
  migrations:
    - name: backfill-2-0
      to: "2.0.0"               # runs once when the deployed version crosses 2.0.0
      stage: web
      actions:
        - name: backfill
          job:
            sourceRef:
              name: web-migrations
  stages:
    - name: web
      sourceRef:
        apiVersion: jaas.metio.wtf/v1
        kind: JsonnetSnippet
        name: web

Let the version travel with the rendered manifests

Pinning works, but the cleaner pattern is to let the version ride inside the manifests the snippet renders — so a single value flows from your CI all the way to the rollout gate. Feed the version into the snippet and stamp it onto the standard app.kubernetes.io/version label (and the image tag, from the same value):

// web.jsonnet
local version = std.extVar('version');   // supplied by JaaS extVars / your CI
{
  apiVersion: 'apps/v1',
  kind: 'Deployment',
  metadata: {
    name: 'web',
    labels: { 'app.kubernetes.io/version': version },   // ← the version, in the manifest
  },
  spec: {
    template: {
      metadata: { labels: { 'app.kubernetes.io/version': version } },
      spec: { containers: [{ name: 'web', image: 'registry.example/web:' + version }] },
    },
  },
}

Then point version.fromObject at that object and drop the inline value — the controller reads the label off the rendered Deployment:

spec:
  version:
    fromObject:
      stage: web
      kind: Deployment
      name: web
      # defaults to the app.kubernetes.io/version label
  migrations:
    - name: backfill-2-0
      to: "2.0.0"
      stage: web
      actions:
        - name: backfill
          job:
            sourceRef:
              name: web-migrations
  stages:
    - name: web
      sourceRef:
        apiVersion: jaas.metio.wtf/v1
        kind: JsonnetSnippet
        name: web

Now the version has exactly one source of truth — the value your pipeline feeds the snippet — and it shows up in the image tag, the version label, and the migration gate together. The same fromObject works for a GitRepository/OCIRepository source too; only a source that ships a dedicated file wants version.fromArtifact instead. See versioned migrations for all three.

Next

From here, add more stages, pre/post actions, or update windows to turn this single rollout into a gated, multi-stage release. To parameterize per environment, see Parameters.

Prerequisites

• A Kubernetes cluster with kubectl and helm configured against it.
• Flux source-controller, specifically the ExternalArtifact API (source.toolkit.fluxcd.io). A StageSet stage always resolves to an ExternalArtifact, so the CRD must exist. ExternalArtifact lands in Flux v2.7.0; install at least that version. The controller also watches GitRepository, OCIRepository, and Bucket sources for producer-aware resolution.
• cert-manager, only if you choose the cert-manager webhook certificate mode. The chart defaults to self-signed, which provisions and rotates the admission webhook’s TLS in-process and needs no cert-manager. See production for the trade-off.

JaaS, JOI, or any particular artifact producer are not required to install the controller — those are sources of ExternalArtifacts, wired up per StageSet.

Install with Helm

The controller is distributed as an OCI Helm chart. The deployment manifests live in the chart, not in the controller repository.

helm upgrade --install stageset-controller \
  oci://ghcr.io/metio/helm-charts/stageset-controller \
  --namespace stageset-system --create-namespace

The container image is ghcr.io/metio/stageset-controller; the chart pins the tag to its own appVersion by default.

Every setting referenced across these docs — HA replicas, the rollback store, webhook mode, NetworkPolicy, the ServiceMonitor, and the rest — is a Helm value. The chart’s README and values.yaml document the full, current list.

What the chart installs

• The controller Deployment, its ServiceAccount, and the cluster RBAC it needs (a ClusterRole + ClusterRoleBinding, plus a namespaced leader-election Role/RoleBinding).
• The CRDs — StageSet and StageInventory.
• The validating admission webhook (ValidatingWebhookConfiguration + a webhook Service).
• A metrics Service (and an opt-in ServiceMonitor).
• The Flagger gate Service for the read-only stage-gate endpoint.
• Opt-in extras: NetworkPolicy, PodDisruptionBudget, HorizontalPodAutoscaler, a rollback-store PersistentVolumeClaim, and a managed Namespace.

About the CRDs

The CRDs ship inside the chart’s regular templates (not Helm’s special crds/ directory), so a helm upgrade applies schema changes like any other resource. This is governed by crds.create (default true). The CRDs carry helm.sh/resource-policy: keep, so a helm uninstall leaves them — and your StageSets — in place; remove them by hand if you really mean to.

If you manage CRDs out of band, the raw definitions are also published in the controller repository under config/crd/ and can be applied with kubectl apply --server-side -f.

Verify

kubectl -n stageset-system get deploy stageset-controller
kubectl get crd stagesets.stages.metio.wtf stageinventories.stages.metio.wtf

Once the controller is Available, create your first StageSet.

Symptom

READY=False, REASON=InvalidSpec. The Message names the offending field or action. Terminal: the controller does not requeue until the spec changes.

Cause

The spec failed validation that the CRD schema cannot express cheaply, normally one of:

• an action sets zero or more than one verb — each action must set exactly one of patch, http, wait, job, delete, apply (see actions);
• spec.migrations without spec.version, or a migration anchored to a stage name that does not exist (see versioned migrations);
• spec.version does not name exactly one source — set one of value, fromObject, or fromArtifact;
• spec.decryption.provider is not sops, or a secretRef is given without a name (see encryption);
• an invalid update window — a malformed schedule, duration, or timeZone (see update windows).

The admission webhook normally rejects these at write time; seeing this on the object means the webhook was bypassed or disabled and the reconciler caught it.

Diagnosis

kubectl describe stageset <name> -n <namespace>

Read the Message — it names the stage, action, or field.

Remediation

Fix the spec per the Message:

• give each action exactly one verb;
• set spec.version (to one of value/fromObject/fromArtifact) whenever spec.migrations is present, and anchor each migration to a real stage;
• set exactly one spec.version source;
• use provider: sops for spec.decryption, with a named secretRef when one is given;
• correct any malformed update window (schedule, duration, timeZone).

If the webhook should have caught this, confirm the ValidatingWebhookConfiguration is installed and its service is reachable.

Symptom

READY=False, REASON=InvalidVersion. Terminal: the run does not requeue until the spec or the version file is fixed.

Cause

A version spec.version (or a migration boundary) could not be resolved to a parseable semver. The controller refuses to proceed rather than deploy a half-versioned system — a system whose recorded version is unknown is worse for migrations than an unversioned one. The Message names which input failed. By version source:

• spec.version.value — the inline string is not a semver.
• spec.version.fromObject — the named stage doesn’t exist; the object (kind/name) isn’t in the stage’s rendered manifests; the fieldPath is invalid JSONPath or resolves to empty; or the value read (by default the app.kubernetes.io/version label) is missing or not a semver.
• spec.version.fromArtifact — the named stage doesn’t exist; the file at path is missing from the stage’s artifact, empty, or doesn’t parse as a semver.
• spec.version sets none of value/fromObject/fromArtifact.
• A migration’s to or from is not a valid semver.
• The recorded status.version is not a semver (corrupted status).

Common triggers across all of them: a v prefix or trailing whitespace the parser rejects, or non-semver text (e.g. a Git SHA or a latest tag) where a version was expected.

Diagnosis

kubectl describe stageset <name> -n <namespace>   # Message names the failing input
kubectl get stageset <name> -n <namespace> -o jsonpath='{.spec.version}{"\n"}'

Then, depending on which source the Message names:

# fromObject: confirm the field carries a bare semver on the rendered object
stagesetctl build <name> -n <namespace> --stage <stage> | grep -i version

# fromArtifact: confirm the file exists and contains only a semver (e.g. 2.1.0)
# inspect the resolved artifact for the stage named in the Message

Remediation

Match the failing input in the Message:

• value — correct the inline string to a bare semver (2.1.0, not v2.1.0).
• fromObject — fix the stage/kind/name to point at a real rendered object, fix the fieldPath, and ensure the field (default: app.kubernetes.io/version label) carries a semver.
• fromArtifact — fix path/stage to the real version file, or correct the file to a bare semver.
• migration to/from — correct the boundary to a valid semver.
• If you don’t need versioned migrations, remove spec.version entirely (this disables versioning and migrations).

There are two ways to run the controller, and they map onto two different trust models. Pick the one that matches your cluster:

• Multi-tenant — the controller holds no write access of its own and applies every StageSet impersonating that StageSet’s serviceAccountName. Each tenant’s RBAC bounds what its releases can touch. This is the chart default.
• Single-tenant — the cluster has one operator, so per-tenant isolation buys nothing. Run the controller under its own identity bound to cluster-admin and skip impersonation entirely — the model Flux’s helm-controller uses in its default install.

The two sections below set each one up. The optional watch scoping narrows which namespaces a multi-tenant controller sees.

Impersonation (multi-tenant)

The controller never applies your manifests as itself. Set serviceAccountName and every operation for that StageSet — build, apply, prune, actions — is performed impersonating that ServiceAccount. The StageSet can do exactly what the SA’s RBAC permits, and nothing more.

spec:
  serviceAccountName: payments-deployer    # all writes impersonate this SA
  stages:
    - name: app
      sourceRef:
        name: payments-app

Grant the SA only the rights that release needs:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: payments-deployer
  namespace: payments
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: edit
subjects:
  - kind: ServiceAccount
    name: payments-deployer
    namespace: payments

This is the multi-tenancy model: isolation comes from each StageSet being bounded by its tenant SA, not from the controller’s own grant — by default the chart gives the controller impersonate and read access, no blanket write. A StageSet with no serviceAccountName, or one bound to a too-narrow SA, fails closed rather than escalating.

Single-tenant cluster-admin

On a cluster with a single operator, per-StageSet impersonation is friction with no payoff — there is no other tenant to isolate from. Run the controller the way Flux’s helm-controller runs by default: under its own ServiceAccount, bound to the built-in cluster-admin ClusterRole. StageSets then omit serviceAccountName and apply as the controller, which can write any kind cluster-wide.

Turn it on with one Helm value:

rbac:
  clusterAdmin: true     # bind the controller SA to cluster-admin

helm upgrade --install stageset-controller \
  oci://ghcr.io/metio/helm-charts/stageset-controller \
  -n stageset-system --create-namespace \
  --set rbac.clusterAdmin=true

StageSets then need nothing tenancy-related — they apply directly:

apiVersion: stages.metio.wtf/v1
kind: StageSet
metadata:
  name: platform
  namespace: stageset-system
spec:
  stages:
    - name: app
      sourceRef:
        name: platform-app    # applied by the controller's cluster-admin identity

When serviceAccountName is unset and no kubeConfig is given, the controller applies with its own client — so the cluster-admin binding is what lets those StageSets write. The trade-off: every StageSet on the cluster has full write access, so this is for single-tenant clusters only. Leave rbac.clusterAdmin at its default false and use impersonation whenever more than one team shares the cluster. The two mix — a cluster-admin controller still honors serviceAccountName on any StageSet that sets it, dropping to that SA’s rights for that release.

Scoping the controller to a namespace set

By default the controller watches every namespace. To run one controller per tenant-group instead — disjoint deployments that each see only their own namespaces — set controller.watchNamespaces:

controller:
  watchNamespaces:
    - team-a
    - team-b

This does two things together:

• Cache scoping. The manager’s informers only observe StageSets and sources in the listed namespaces. Resources elsewhere never enter the cache, so the controller cannot act on them even if RBAC would allow it.
• RBAC pivot. The chart stops binding the tenant ClusterRole cluster-wide and instead renders one RoleBinding per listed namespace — defense in depth, so the apiserver also refuses out-of-scope calls. (The cluster-scoped webhook-caBundle grant stays a ClusterRoleBinding, since a ValidatingWebhookConfiguration is not namespaced.)

Run several releases with disjoint watchNamespaces lists to shard the cluster across independent controller instances. Combine it with impersonation for the tightest setup: each instance sees only its namespaces, and each StageSet is bounded by its tenant SA.

Remote clusters

Point a StageSet at another cluster with kubeConfig, referencing a Secret that holds a kubeconfig. Combined with serviceAccountName, the controller applies to the remote cluster as the impersonated identity there.

spec:
  serviceAccountName: payments-deployer
  kubeConfig:
    secretRef:
      name: prod-eu-kubeconfig
      # key defaults to "value" (the Flux convention); set it to override
  stages:
    - name: app
      sourceRef:
        name: payments-app

The Secret is read with the controller’s own identity — connecting to the target cluster is the controller’s job — and the kubeconfig payload defaults to the value key. A self-contained kubeconfig is required; configMapRef-style cloud-provider auth is not supported.

Cross-namespace sourceRef and dependsOn references can be disabled cluster-wide with the controller’s --no-cross-namespace-refs flag when you want hard namespace isolation.

Metrics

The controller registers custom metrics on the controller-runtime registry, served on --metrics-bind-address (:8080) alongside the standard controller_runtime_* and workqueue_* series. Enable scraping with the chart’s opt-in ServiceMonitor (metrics.serviceMonitor.enabled):

# values.yaml
metrics:
  serviceMonitor:
    enabled: true        # needs the Prometheus operator CRDs

Alerts

The chart ships an opt-in PrometheusRule with a starter alert set, gated on metrics.prometheusRule.enabled (requires the Prometheus operator CRDs). It covers the custom stageset_* metrics plus controller-runtime signals:

Every threshold is a knob under metrics.prometheusRule.thresholds, and extraAlertLabels is merged onto every rendered alert so all stageset alerts can route through one Alertmanager receiver. Each alert carries a runbook_url annotation pointing at the matching runbook page on this site (metrics.prometheusRule.runbookBaseURL); the reconcile-errors alert templates the URL on $labels.reason. Append your own rules under metrics.prometheusRule.extraRules, and silence a built-in alert by raising its threshold rather than forking the chart.

Events

The controller emits Kubernetes Events on every Ready-condition transition, so kubectl describe stageset <name> and Flux’s notification-controller (via an Alert targeting kind: StageSet) both surface what happened. Normal events include Succeeded, UpdateDeferred, MigrationStarted, and MigrationCompleted; warnings include StageFailed, DriftCorrected, RolledBack, MigrationFailed, OnFailureAction, and RollbackStoreFailed.

Runbooks

Every actionable Ready-condition reason has a runbook covering the symptom, cause, diagnosis, and remediation. Set --runbook-base-url (the chart’s controller.runbookBaseURL, which defaults to this docs site) to a published copy of those pages and the controller appends (runbook: <base>/<reason>/) to the Ready message (the reason lower-cased into a path segment), so a kubectl describe links straight to the fix. Healthy reasons (Succeeded, Suspended) get no link.

# values.yaml — point at your own mirror, or set "" to drop the links
controller:
  runbookBaseURL: https://runbooks.internal/stageset

For example, a StageFailed StageSet then shows:

Message:  stage "application" failed: … (runbook: https://runbooks.internal/stageset/stagefailed/)

Forcing a reconcile

The controller reconciles on its spec.interval, on source changes, and on demand. To trigger an out-of-band run, stamp the standard annotation — which is what flux reconcile and stagesetctl reconcile do for you:

kubectl annotate stageset my-app \
  reconcile.fluxcd.io/requestedAt="$(date -u +%FT%TZ)" --overwrite

The handled token is recorded in status.lastHandledReconcileAt.

Drift correction

On a steady-state reconcile the controller re-asserts the desired state, healing out-of-band changes to managed objects. Each correction emits a DriftCorrected event and increments stageset_drift_corrected_total. Tighten the cadence with spec.driftDetectionInterval when you need faster healing than spec.interval.

A rollout takes parameters at two distinct layers, which serve different purposes:

• Render-time parameters (JaaS). Change what gets rendered. The Jsonnet computes its output from top-level arguments (tlas) and external variables (externalVariables). Different values produce a different ExternalArtifact.
• Delivery-time parameters (StageSet postBuild). Inject values into already-rendered manifests, per stage, by string substitution — the same mechanism Flux’s kustomize-controller uses.

Use render-time parameters for structural logic; use delivery-time parameters to stamp environment-specific values onto a shared artifact.

Render-time: JaaS TLAs and external variables

Top-level arguments map to a Jsonnet function(...):

// main.jsonnet
function(name='web', replicas='2')
  { apiVersion: 'apps/v1', kind: 'Deployment', metadata: { name: name },
    spec: { replicas: std.parseInt(replicas) /* … */ } }

apiVersion: jaas.metio.wtf/v1
kind: JsonnetSnippet
metadata:
  name: web
  namespace: apps
spec:
  sourceRef: { kind: GitRepository, name: web-manifests, path: ./jsonnet }
  tlas:                          # → function(name, replicas)
    name: ["web"]
    replicas: ["3"]
  externalVariables:            # → std.extVar('environment')
    environment: "production"

tlas is a map of name → list of values (a single-element list for a scalar argument; multiple values become a JSON array). externalVariables are plain strings read with std.extVar.

Delivery-time: StageSet postBuild substitution

When the rendered manifests carry ${var} placeholders, a stage substitutes them at apply time — from inline values, ConfigMaps, and Secrets:

apiVersion: stages.metio.wtf/v1
kind: StageSet
metadata:
  name: web
  namespace: apps
spec:
  stages:
    - name: web
      sourceRef:
        apiVersion: jaas.metio.wtf/v1
        kind: JsonnetSnippet
        name: web
      postBuild:
        substitute:
          cluster_name: prod-eu
        substituteFrom:
          - kind: ConfigMap
            name: cluster-vars
          - kind: Secret
            name: cluster-secrets
            optional: true

A manifest field like value: "${cluster_name}" becomes value: "prod-eu" for this stage.

Reusing one artifact across environments

The two layers combine into a common pattern: render an environment-agnostic artifact once with JaaS, then have several StageSets — one per environment — consume that same artifact and stamp their own values with postBuild:

# staging
spec:
  stages:
    - name: web
      sourceRef: { apiVersion: jaas.metio.wtf/v1, kind: JsonnetSnippet, name: web }
      postBuild:
        substituteFrom:
          - { kind: ConfigMap, name: staging-vars }
---
# production (same artifact, different values)
spec:
  stages:
    - name: web
      sourceRef: { apiVersion: jaas.metio.wtf/v1, kind: JsonnetSnippet, name: web }
      postBuild:
        substituteFrom:
          - { kind: ConfigMap, name: production-vars }

One render, many environments — each StageSet bounded by its own ServiceAccount and gated by its own actions and update windows.

Symptom

READY=False, REASON=PreviousRevisionUnavailable. The StageSet has spec.rollbackOnFailure set, a run failed, and the controller could not restore the last-good revisions.

Cause

rollbackOnFailure restores the previously-applied artifact revisions by re-fetching their recorded URLs and verifying their digests. That only works while the producer still retains those revisions. This reason means a revision the rollback needs is no longer fetchable — the producer garbage-collected it.

Rollback is best-effort by contract: it works exactly when producers retain. Common triggers:

• a JaaS JsonnetSnippet with spec.history: 1 (the default) — only the current revision is kept, so there is no previous revision to roll back to
• a stock source-controller source, which retains only the current revision
• the previous revision aged out of the producer’s retention window
• the run used SOPS decryption and the key Secret was rotated or deleted — rollback re-runs decryption rather than restoring plaintext, so it fails closed when the key is gone, even for a revision the rollback store holds

Diagnosis

kubectl describe stageset <name> -n <namespace>   # Message names the stage + revision

Check the producer’s retention. For a JaaS snippet:

kubectl get jsonnetsnippet <name> -n <namespace> -o jsonpath='{.spec.history}'

Remediation

The cluster is left at the partially-applied failed state; resolve the underlying failure (see the failing stage’s own runbook) and fix forward — the StageSet converges once the desired revision applies cleanly.

To make rollback reliable in future, either:

• Increase producer retention so at least one previous revision is always fetchable — JaaS snippets used with rollbackOnFailure should set spec.history: 2 (or more); sources that retain only the current revision cannot support the re-fetch path, so rely on source revert instead.
• Configure the external rollback store — a filesystem/RWX PVC (--rollback-store-path) or an S3 bucket (--rollback-store-s3-*); see operations. When the controller pushes rendered output to a store it owns, rollback is bit-exact and independent of producer retention — this PreviousRevisionUnavailable state cannot occur for runs the store holds, unless the run used SOPS and the key Secret is no longer readable (see the SOPS trigger above).

Stages and sources covers the two direct routes — an ExternalArtifact (the default sourceRef.kind) or a Flux GitRepository/OCIRepository/Bucket. This page covers the third: naming the thing that produces an artifact and letting the controller find it. This is useful when an operator publishes an ExternalArtifact from a custom resource (for example JaaS rendering Jsonnet).

Referencing a producer

Set kind (and apiVersion) to a producer resource, and the controller resolves it to the ExternalArtifact that producer publishes — the one whose spec.sourceRef back-references the producer (matched on group, kind, and name). For example, a JaaS JsonnetSnippet renders Jsonnet and publishes an ExternalArtifact; reference the snippet and the controller follows the link:

spec:
  stages:
    - name: dashboards
      sourceRef:
        apiVersion: jaas.metio.wtf/v1
        kind: JsonnetSnippet
        name: grafana-dashboards

The controller also watches the common Flux source kinds (GitRepository, OCIRepository, Bucket) so a stage re-reconciles when an upstream source changes.

Related projects

JOI, JaaS, and StageSet compose end to end:

• JOI publishes Jsonnet libraries as single-layer OCI images (usable both as image-volume mounts and as Flux OCIRepository sources).
• JaaS evaluates Jsonnet — optionally importing those JOI libraries — and publishes the rendered JSON as an ExternalArtifact.
• StageSet references the JsonnetSnippet (or its artifact) and rolls the result out in ordered, gated stages.

Each project is independently useful; a stage reads straight from a GitRepository, OCIRepository, or Bucket, or from any ExternalArtifact regardless of what produced it.

High availability

The controller supports leader-elected HA. Enable leader election and run more than one replica; only the lease holder reconciles, while every replica answers admission webhook calls (admission must stay available even on non-leaders).

• Leader election is toggled with --leader-elect. The binary defaults it to false, but the Helm chart enables it by default (controller.leaderElect: true), so a default install is already lease-guarded even at one replica.
• The lease is named stageset-controller.stages.metio.wtf and lives in the controller’s namespace. It uses controller-runtime’s default timing (~15 s lease duration). The lease is not released eagerly on shutdown, so after a rolling update the new leader takes over when the old lease expires — budget a few seconds of reconcile pause on restart (admission and the gate endpoint are unaffected).
• Scaling: when the chart’s replicas.max exceeds replicas.min it renders a HorizontalPodAutoscaler (CPU target 80%) and a PodDisruptionBudget (minAvailable: 1). At the default 1/1 it sets neither and leaves spec.replicas unmanaged.

The controller watches every namespace by default. Multi-tenancy is enforced per StageSet through impersonation (see below). You can additionally scope the controller to a namespace set with controller.watchNamespaces — one controller instance per tenant-group — and run it under cluster-admin for single-tenant clusters; both are covered in multi-cluster and tenancy.

Hardening

Each option below is shown as the Helm values that configure it. Several are already the chart’s defaults, shown so you can see what is applied and override it for a stricter policy.

Tenant impersonation

The controller never applies your manifests with its own identity. Every cluster operation for a StageSet — building, applying, pruning, running actions — is performed impersonating the StageSet’s spec.serviceAccountName (the chart grants the controller impersonate, not write access). A StageSet can only do what its tenant SA permits; an over-broad or missing SA fails closed.

This one lives on the StageSet, not in the chart — give every production StageSet a scoped ServiceAccount:

apiVersion: stages.metio.wtf/v1
kind: StageSet
metadata: { name: payments, namespace: payments }
spec:
  serviceAccountName: payments-deployer   # scoped to exactly this release's needs
  # …

Pod security context

The chart runs a non-root, read-only-root-filesystem pod with all capabilities dropped, on a gcr.io/distroless/static:nonroot image (no shell or package manager). These are the rendered defaults:

podSecurityContext:
  runAsNonRoot: true
  seccompProfile:
    type: RuntimeDefault
securityContext:
  runAsNonRoot: true
  runAsUser: 65532
  runAsGroup: 65532
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities:
    drop: [ALL]
  seccompProfile:
    type: RuntimeDefault

Resource limits

Requests equal limits, so the pod is fully constrained:

resources:
  cpu: 50m
  memory: 256Mi
  ephemeralStorage: 32Mi   # /tmp and the self-signed cert dir are emptyDirs

Pod-Security Standards namespace

Have the chart create the install namespace with restricted PSS labels:

namespace:
  create: true
  pssLevel: restricted     # or: baseline / privileged

Network policy

The gate endpoint is unauthenticated (read-only GET /gate/{namespace}/{stageset}/{stage}). Turn on the ingress-only NetworkPolicy to fence it — and the webhook/metrics ports — to only the peers that need them:

networkPolicy:
  enabled: true            # admits the webhook (9443), metrics (8080), gate (8082)

The policy is ingress-only, so it does not restrict egress — the controller can still fetch stage artifacts over HTTP from source-controller (an ExternalArtifact or a GitRepository/OCIRepository/Bucket is served from the same artifact endpoint). If your cluster default-denies egress, add an egress allowance to source-controller (and DNS) so those fetches succeed.

Admission webhook TLS

webhook.certMode chooses how the webhook serving certificate is obtained:

webhook:
  certMode: cert-manager   # cert-manager issues + rotates the cert (requires cert-manager)
  # certMode: self-signed  # chart default: in-pod CA + serving cert, rotated at
  #                          validity/3, with no cert-manager dependency

Reference setups

Two HA shapes — on-prem with shared RWX storage, and AWS/EKS with S3 — over the same backbone: a leader-elected pair (or trio), a rollback store reachable from whichever pod holds the lease, cert-manager for the webhook, a NetworkPolicy fencing the unauthenticated gate, and a ServiceMonitor if you run Prometheus.

Both run two replicas for HA (replicas.max above replicas.min also renders a PDB and an HPA) and set webhook.certMode: cert-manager, so cert-manager must be installed in the cluster.

On-prem (RWX storage)

The rollback store gives bit-exact rollbacks that outlive producer GC. With HA replicas it must be reachable from whichever pod holds the lease, so use a ReadWriteMany PVC on your on-prem storage class — every replica mounts the same volume.

# values-onprem.yaml
replicas:
  min: 2                 # leader-elected HA; the non-leader still serves admission
  max: 3                 # > min renders an HPA (CPU 80%) and a PodDisruptionBudget

controller:
  leaderElect: true

rollbackStore:
  backend: pvc
  pvc:
    accessModes: [ReadWriteMany]
    storageClass: nfs-client     # your RWX class (NFS, CephFS, …)
    size: 10Gi

webhook:
  certMode: cert-manager         # requires cert-manager in the cluster

networkPolicy:
  enabled: true                  # fences the unauthenticated gate endpoint

metrics:
  serviceMonitor:
    enabled: true

helm upgrade --install stageset-controller \
  oci://ghcr.io/metio/helm-charts/stageset-controller \
  --namespace stageset-system --create-namespace \
  -f values-onprem.yaml

AWS / EKS (S3)

On EKS, back the rollback store with S3 and let the controller assume an IAM role through IRSA — no static keys. Annotate the controller’s ServiceAccount with the role ARN and leave the S3 credentials empty; the store’s minio-go client picks the role up from the pod’s web-identity token.

# values-eks.yaml
replicas:
  min: 2
  max: 3

controller:
  leaderElect: true

serviceAccount:
  annotations:
    # an IAM role granting s3:GetObject/PutObject/ListBucket/DeleteObject on the bucket
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/stageset-controller

rollbackStore:
  backend: s3
  s3:
    endpoint: s3.eu-west-1.amazonaws.com
    bucket: my-org-stageset-rollback
    region: eu-west-1
    # no existingSecret → credentials come from the IRSA role above

webhook:
  certMode: cert-manager

networkPolicy:
  enabled: true

metrics:
  serviceMonitor:
    enabled: true

helm upgrade --install stageset-controller \
  oci://ghcr.io/metio/helm-charts/stageset-controller \
  --namespace stageset-system --create-namespace \
  -f values-eks.yaml

Alongside the other Flux controllers

stageset-controller is a Flux citizen and needs no special wiring to coexist with source-controller, kustomize-controller, helm-controller, and notification-controller. It reads ExternalArtifact (and the standard GitRepository, OCIRepository, and Bucket sources) from source-controller, and notification-controller routes its events through an Alert that targets kind: StageSet — no Provider/Alert plumbing of its own. Install it in its own namespace (e.g. stageset-system) next to flux-system; the only cluster-scoped pieces are its CRDs, ClusterRole, and webhook configuration.

Alongside JaaS

JaaS renders Jsonnet and publishes the result as an ExternalArtifact, which is what a StageSet stage consumes — so the two compose directly. Reference the artifact by name, or name the producing JsonnetSnippet and let stageset-controller resolve it (see producer-aware sources). They can share a cluster and namespace or stay separate; both are reconciled by Flux and both apply under per-tenant impersonation, so the security model is consistent end to end.

Settings you can tune

The chart wires the controller; you set Helm values. The set worth thinking about is below — each row is the value, its default, and when you’d change it. Everything else the chart configures for you (see what the chart manages).

Every option is set the same way — in your values file, applied with helm upgrade --install … -f values.yaml. The reference setups above are complete, copy-pasteable examples.

What the chart manages

You do not configure these — the chart wires them so the controller behaves correctly out of the box:

• Leader election and HA plumbing — the lease, and the PDB/HPA when replicas.max > replicas.min.
• The admission webhook — the server, its Service, the ValidatingWebhookConfiguration, and the certificate (cert-manager Certificate or the in-pod self-signed renewer, per webhook.certMode).
• Endpoints — metrics, health probes, and the gate, on their Services.
• RBAC — the ClusterRole/bindings the controller needs, including the impersonate verb (it never applies as itself).
• A hardened pod — non-root, read-only root filesystem, dropped capabilities, seccomp RuntimeDefault (see pod security context).
• Per-tenant impersonation — every apply runs as the StageSet’s spec.serviceAccountName.

Controller flags reference

You never pass these directly — the chart sets them from your Helm values and its own defaults — but it helps to know what each does and which value drives it.

The PVC and S3 rollback stores are mutually exclusive — the chart enforces it via rollbackStore.backend. The controller also accepts controller-runtime’s --zap-* logging flags.

StageSet integrates with both progressive-delivery controllers: Flagger and Argo Rollouts. The controller exposes a read-only gate endpoint and a readiness gauge so either one can hold a promotion until a StageSet stage is healthy; ready checks let a stage wait on a Rollout in return. Pick the section for your controller below — see also StageSet vs Argo Rollouts.

The gate contract

The gate endpoint backs the Flagger integration and the Argo Rollouts JSON-metric option.

GET /gate/{namespace}/{stageset}/{stage}
  200  — the stage is Ready at the currently pinned revision
  403  — the stage is not Ready (or not found / not gateable)

It is served on --gate-bind-address (default :8082) and exposed by the chart’s stageset-controller-gate Service (gate.enabled, on by default). The endpoint is unauthenticated and read-only, so fence it with a NetworkPolicy (production) to admit only your delivery controller.

Flagger

Add a confirm-promotion (or confirm-rollout) webhook to a Flagger Canary pointing at the gate. Flagger blocks the promotion until the gate returns 200:

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: web
  namespace: apps
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web
  analysis:
    interval: 1m
    threshold: 5
    stepWeight: 10
    maxWeight: 50
    webhooks:
      - name: stageset-stage-ready
        type: confirm-promotion
        # gate this canary's promotion on a StageSet stage being Ready
        url: http://stageset-controller-gate.stageset-system:8082/gate/apps/web/web

This is independent of the Flagger strategy: the same webhook gates a weighted canary, an A/B test (header/cookie routing), or a blue-green promotion — the gate only answers “is this stage Ready,” and Flagger decides what to do with that answer.

This coordinates two moving parts: Flagger shifts traffic to a new version only once a StageSet stage that applied the supporting config (a CRD, a migration, a sibling component) reports Ready.

Argo Rollouts

Argo Rollouts gates on analysis metrics (a query that returns a value to compare) rather than a webhook’s HTTP status, so the controller meets it on its own terms in two ways.

Gate on the readiness gauge (recommended)

The controller exports stageset_stage_ready{namespace,stageset,stage} (1 when the stage is Ready, 0 otherwise). Argo’s Prometheus metric provider gates on it directly — no gate endpoint, no Job:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: stageset-stage-ready
  namespace: apps
spec:
  metrics:
    - name: stage-ready
      successCondition: result == 1
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: max(stageset_stage_ready{namespace="apps",stageset="web",stage="web"})

Gate on the JSON endpoint

The same gate endpoint also answers JSON when asked (Accept: application/json), returning {"ready": true, …} with a 200 so Argo’s web metric can parse it (Argo treats a non-2xx as an error, so readiness has to live in the body):

spec:
  metrics:
    - name: stage-ready
      successCondition: "result.ready == true"
      provider:
        web:
          url: http://stageset-controller-gate.stageset-system:8082/gate/apps/web/web
          headers:
            - key: Accept
              value: application/json
          jsonPath: "{$}"

A Job-based metric (curl -fsS … against the gate, succeeding only on 200) is the fallback when the analysis has no Prometheus or web access.

The reverse direction: gate a StageSet on a Rollout

The coordination also works the other way. Because ready checks accept CEL, a StageSet stage can wait on an Argo Rollout finishing its own progressive rollout before the next stage runs:

readyChecks:
  exprs:
    - apiVersion: argoproj.io/v1alpha1
      kind: Rollout
      current: "status.phase == 'Healthy'"
      inProgress: "status.phase in ['Progressing', 'Paused']"
      failed: "status.phase == 'Degraded'"

So StageSet can gate Argo (via the gauge/gate) and Argo’s outcome can gate StageSet (via ready checks) — pick whichever direction your release needs.

Ready checks decide when a stage is healthy enough to let the next stage start. They are purely observational — the controller waits and reports, but takes no action (active steps are actions).

By default, with no readyChecks block, the controller waits for every object the stage applied to report ready via kstatus. readyChecks lets you narrow that to specific objects (checks), add custom health for resources kstatus doesn’t understand (exprs, CEL), bound the wait (timeout), or skip it entirely (disableWait). checks and exprs may be set together.

Explicit objects

Wait for named objects only — useful when a stage applies many objects but only a few gate the next stage:

spec:
  stages:
    - name: infrastructure
      sourceRef:
        name: platform
      readyChecks:
        timeout: 5m
        checks:
          - apiVersion: apiextensions.k8s.io/v1
            kind: CustomResourceDefinition
            name: ledgers.payments.example
          - apiVersion: apps/v1
            kind: Deployment
            name: ledger-operator
            namespace: platform-system

Custom health with CEL

For custom resources kstatus doesn’t understand, describe readiness with CEL expressions. The shape matches kustomize-controller’s healthCheckExprs, so expressions are portable.

      readyChecks:
        exprs:
          - apiVersion: db.example/v1
            kind: Database
            current: "status.phase == 'Running'"
            inProgress: "status.phase in ['Pending', 'Provisioning']"
            failed: "status.phase == 'Failed'"

Opting out

To apply a stage without waiting for readiness (fire-and-forget), disable the wait:

      readyChecks:
        disableWait: true

Symptom

controller_runtime_reconcile_time_seconds p99 for controller="stageset" exceeds the configured threshold; the StageSetReconcileLatencyHigh alert fires (see operations for the alert set and its thresholds).

Cause

A single reconcile does a lot of work — resolve and fetch every stage’s artifact, kustomize-build, server-side apply, prune, verify readiness, and run actions — all impersonating the tenant ServiceAccount. Latency climbs when any of those is slow:

• large artifacts or slow artifact servers,
• many objects per stage (apply + prune scale with object count),
• readiness waits and wait/http/job actions with long timeouts,
• apiserver or tenant-authorization slowness.

Diagnosis

kubectl -n stageset-system logs deploy/stageset-controller --tail=200 | grep -i 'slow\|timeout\|took'

Break the latency down by stage count and artifact size; a single StageSet with many large stages dominates p99.

Remediation

• Split a very large StageSet into smaller ones, or fewer objects per stage.
• Tighten action timeouts so a slow gate fails fast instead of stretching the reconcile.
• Raise spec.interval where freshness isn’t critical.
• Address upstream artifact-server or apiserver latency.

If the queue itself is backing up, see workqueue saturation.

Symptom

READY=False, REASON=ResolveFailed. The Message describes why resolution failed.

Cause

A stage’s sourceRef could not be resolved to an ExternalArtifact for a spec/config or API reason (distinct from “not published yet”, which is SourceNotReady, and “no such object”, which is ArtifactNotFound). Common cases:

• an ambiguous producer — more than one ExternalArtifact back-points at the same producer object, so the target is undefined;
• a cross-namespace ref rejected by --no-cross-namespace-refs;
• an API error reading the source or artifact (RBAC denial, the artifact CRD not installed).

When the failing sourceRef targets another namespace, the Message is deliberately scrubbed to cross-namespace <kind> %q is not reachable so tenants cannot fingerprint other namespaces — check that source CR’s status in its own namespace.

Diagnosis

kubectl describe stageset <name> -n <namespace>
# Ambiguity: are there multiple artifacts pointing at the producer?
kubectl get externalartifact -n <namespace> -o yaml | grep -A3 sourceRef

Remediation

• Ambiguous producer: ensure exactly one ExternalArtifact back-points at the producer, or reference the ExternalArtifact directly by name.
• Cross-namespace rejected: move the source into the StageSet’s namespace, or run the controller without --no-cross-namespace-refs if your tenancy model allows it.
• RBAC / missing CRD: grant the controller (or the impersonated serviceAccountName) read on the source kind, or install the source-controller CRDs.

When a run fails, the controller can restore the last successfully-applied artifact revisions instead of leaving you on a broken release. Rollback is opt-in and needs somewhere to keep prior revisions.

Enabling it

spec:
  rollbackOnFailure: true
  stages:
    - name: app
      sourceRef:
        name: my-app

On a failed run the controller restores each stage’s last-good artifact revision, best-effort, and emits a RolledBack event. The coordinates it restores from are recorded in status.lastAppliedSnapshot.

The rollback store

Rollback needs the prior revision to still be fetchable, so the controller keeps a copy in a rollback store. Configure one on the controller (cluster-wide), via either a shared filesystem or S3:

# filesystem (an RWX PersistentVolumeClaim)
--rollback-store-path=/var/lib/stageset/rollback

# or S3-compatible object storage
--rollback-store-s3-endpoint=s3.example.com
--rollback-store-s3-bucket=stageset-rollback

The two are mutually exclusive. With no store configured, rollback can only use a prior revision the producer itself still retains; a dedicated store makes rollback reliable across producer pruning.

Encryption at rest

The store keeps each stage’s rendered output, which includes any Secret’s data — including SOPS-decrypted values (see secrets encryption). Treat it as sensitive and keep it encrypted at rest:

• S3 encrypts by default. --rollback-store-s3-sse (chart: rollbackStore.s3.sse) is s3 (SSE-S3) out of the box; set kms with rollbackStore.s3.sseKmsKeyId for SSE-KMS, or none only for a backend that cannot honor an SSE header. A rejected SSE write is non-fatal — it warns via a RollbackStoreFailed event and skips the store write; the rollout still succeeds.
• Filesystem can’t encrypt itself — back the PVC with an encrypted volume (an encrypted StorageClass, LUKS, or cloud-disk encryption). The controller logs a reminder at startup when the file store is enabled.

If a restore can’t proceed because the previous revision is gone, the run fails with the PreviousRevisionUnavailable reason (see its runbook), and a store problem surfaces as a RollbackStoreFailed event.

A stage’s source can carry SOPS-encrypted files — typically a Secret whose values are encrypted — and the controller decrypts them in memory, before building and applying the manifests. This mirrors Flux’s kustomize-controller decryption contract, so an existing SOPS-encrypted repository works unchanged.

Set spec.decryption and point it at a Secret holding the keys:

apiVersion: stages.metio.wtf/v1
kind: StageSet
metadata:
  name: payments
  namespace: payments
spec:
  serviceAccountName: payments-deployer
  decryption:
    provider: sops          # the only provider
    secretRef:
      name: sops-age        # a Secret in this namespace holding the age key
  stages:
    - name: app
      sourceRef:
        kind: GitRepository
        name: payments-config   # contains an encrypted secret.yaml

Walkthrough — age

age is the simplest key type and needs no external service. Take a Secret from plaintext to a GitOps-safe rollout in four steps.

1. Generate an age key. The file holds the private key; the printed age1… line is the public recipient to encrypt to.

age-keygen -o age.agekey
# public key: age1qz…

2. Encrypt a Secret. Encrypt only its values, so the file stays a valid Kubernetes object, then commit secret.enc.yaml (never the plaintext):

# secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: payments-db
  namespace: payments
stringData:
  password: s3cr3t-do-not-commit-plaintext

sops --encrypt --age age1qz… \
  --encrypted-regex '^(data|stringData)$' \
  secret.yaml > secret.enc.yaml

3. Put the private key in the cluster under a .agekey data entry. Store age.agekey itself somewhere safe — it is the only thing that can decrypt the Secret.

kubectl create secret generic sops-age \
  --namespace payments \
  --from-file=keys.agekey=age.agekey

4. Decrypt on rollout. Point a StageSet at the source holding secret.enc.yaml and set spec.decryption (as in the example above). On reconcile the controller fetches the source, decrypts every SOPS file in memory, builds, and applies — so the cluster holds the plaintext payments-db Secret while Git only ever held ciphertext. Grant the deployer ServiceAccount read access to the key Secret (see tenancy below).

Pairing with JaaS-rendered manifests

A realistic app renders its config from Jsonnet with JaaS and keeps only its Secret encrypted. The two compose cleanly because each owns one concern:

• JaaS renders the non-secret manifests. It evaluates Jsonnet server-side and cannot hold secret values: SOPS ciphertext carries a MAC over the whole encrypted document, so it can’t be authored in Jsonnet — and routing plaintext secrets through a render service is what you are avoiding.
• The Secret stays SOPS-encrypted in Git, as in the walkthrough.
• The controller decrypts and orders both under one spec.decryption:

spec:
  serviceAccountName: payments-deployer
  decryption:
    provider: sops
    secretRef:
      name: sops-age
  stages:
    - name: secrets                 # decrypt + apply the SOPS Secret first
      sourceRef:
        kind: GitRepository
        name: payments-secrets
    - name: app                     # then the JaaS-rendered app that mounts it
      sourceRef:
        apiVersion: jaas.metio.wtf/v1
        kind: JsonnetSnippet
        name: payments-app

The secrets stage runs first; only once the Secret is applied does the app stage roll out the rendered manifests that mount it. The encrypted Secret and the rendered config live in separate sources, so the Jsonnet author never touches secret material.

The fields

• provider — the decryption backend. Only sops is supported.
• secretRef.name — a Secret in the StageSet’s namespace holding the keys, using the SOPS conventions: age private keys under data entries ending in .agekey, armored PGP private keys under .asc. Optional — omit it for a cloud-KMS-only setup.

How keys are read — tenancy

The key Secret is read in the StageSet’s namespace under its serviceAccountName, exactly like the manifests it applies. A tenant can only decrypt with key material its own ServiceAccount is allowed to read, so a key in one namespace is never reachable from another. Grant the deployer SA get on the key Secret:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: payments-deployer-sops
  namespace: payments
rules:
  - apiGroups: [""]
    resources: [secrets]
    resourceNames: [sops-age]
    verbs: [get]

In a single-tenant cluster-admin install (no serviceAccountName), the controller reads the key Secret under its own identity instead.

Decryption and the rollback store

Decrypted bytes exist only in memory on the apply path. The one place rendered output is persisted is the optional rollback store, which is encrypted at rest (S3 SSE by default; an encrypted volume for the file store) — so a decrypted Secret never lands in plaintext on disk. See encryption at rest.

A rollback re-fetches the previous source and runs decryption again rather than restoring plaintext, so the key Secret must still exist when a rollback fires. If the key was rotated or deleted in the meantime, the rollback fails closed with PreviousRevisionUnavailable instead of applying a stale or unreadable Secret — an encrypted store cannot avoid this, and it is the safe failure direction.

Cloud KMS

SOPS files encrypted with a cloud KMS key (AWS KMS, GCP KMS, Azure Key Vault, or HashiCorp Vault) decrypt through the controller’s ambient credentials — e.g. an IRSA role on EKS, wired via serviceAccount.annotations. No in-cluster key Secret is needed, so secretRef may be omitted for a KMS-only StageSet:

spec:
  decryption:
    provider: sops          # secretRef omitted; KMS uses the controller's identity

One consequence to weigh in a multi-tenant cluster: unlike age (read under the tenant SA), cloud KMS uses the controller’s identity, so any StageSet can decrypt a file encrypted with a KMS key the controller’s role can access. This matches Flux’s kustomize-controller. Scope the controller’s KMS grant accordingly, or use age keys for hard per-tenant isolation.

What’s supported

• age keys via secretRef — read under the tenant SA. The resource-level pattern (--encrypted-regex '^(data|stringData)$') is the tested path.
• PGP keys via secretRef (.asc entries) — read under the tenant SA, pure Go, no gpg binary or keyring needed. See PGP keys.
• Cloud KMS (AWS/GCP/Azure/Vault) via the controller’s ambient credentials.
• Encrypted files feeding a secretGenerator — an encrypted .env (or other file) referenced by a kustomize secretGenerator is decrypted before the build, so the generated Secret carries the plaintext.
• A file with no SOPS metadata passes through untouched, so encrypted and plain manifests can sit side by side in one source.

PGP keys

PGP works tenant-scoped, like age: put one or more armored private keys in the secretRef Secret under data entries suffixed .asc. The data key is decrypted in pure Go (ProtonMail/go-crypto) directly from those keys — no gpg binary, no GnuPG keyring, and no GNUPGHOME — and the keys are read under the StageSet’s serviceAccountName, so a tenant can only use material its ServiceAccount can read.

# export the armored private key and load it into the key Secret
gpg --export-secret-keys --armor 0xYOURFINGERPRINT > key.asc

kubectl create secret generic sops-keys \
  --namespace payments \
  --from-file=pgp.asc=key.asc

spec:
  decryption:
    provider: sops
    secretRef:
      name: sops-keys      # holds the *.asc private key(s)

One Secret can carry both age (*.agekey) and PGP (*.asc) keys; the right one is used per file. For a fresh setup, age is simpler and the recommended default, but an existing PGP-encrypted repository needs no migration.

Symptom

READY=False, REASON=SourceNotReady. Transient: the controller requeues and clears the condition once the source publishes.

Cause

A stage’s sourceRef resolved to an ExternalArtifact (directly, or via a producer’s RFC-0012 back-pointer such as a JaaS JsonnetSnippet), but that artifact’s status.conditions[Ready] is not yet True — its producer has not finished publishing a revision. The StageSet gates on Ready=True so it never builds against a half-written artifact.

Diagnosis

# Which artifact, and is it Ready?
kubectl get externalartifact -n <namespace>
kubectl describe externalartifact <name> -n <namespace>

# If the producer is a JsonnetSnippet (or other producer kind), check it:
kubectl describe jsonnetsnippet <name> -n <namespace>

Remediation

This usually clears on its own when the producer publishes. If it persists:

• confirm the producing controller (e.g. the JaaS operator, or Flux source-controller) is running and reconciling the producer object;
• check the producer’s own Ready condition for an upstream error (a failed render, an unreachable GitRepository/OCIRepository source);
• once the producer reports Ready=True with a status.artifact, the StageSet converges on the next reconcile.

If the artifact never appears at all, the reason is ArtifactNotFound; a spec/API resolution failure is ResolveFailed. See stages and sources.

A stage resolves its sourceRef to a Flux artifact. You have two routes:

manifests in Git / OCI / Bucket  ──────────────────────────►  StageSet   (direct)
manifests in Git / OCI / Bucket  ──►  a renderer (JaaS)  ──►  ExternalArtifact  ──►  StageSet

Use the direct route when the source already holds ready-to-apply manifests (the same thing Flux’s kustomize-controller consumes). Use the renderer route when you generate manifests first — e.g. evaluating Jsonnet with JaaS.

This page is the copy-pasteable recipe per source kind. For how sourceRef resolution works as a concept — and the path, prune, patches, and postBuild knobs that shape a stage — see stages and sources.

Direct: Git

Point a stage straight at a GitRepository:

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: web-manifests
  namespace: apps
spec:
  interval: 1m
  url: https://github.com/acme/web-manifests
  ref:
    branch: main
---
apiVersion: stages.metio.wtf/v1
kind: StageSet
metadata:
  name: web
  namespace: apps
spec:
  stages:
    - name: web
      sourceRef:
        kind: GitRepository
        name: web-manifests
      path: ./manifests        # build a sub-path of the repo

Direct: OCI

Manifests pushed as an OCI artifact (e.g. with flux push artifact):

apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
  name: web-manifests
  namespace: apps
spec:
  interval: 5m
  url: oci://ghcr.io/acme/web-manifests
  ref:
    tag: "2.1.0"
---
apiVersion: stages.metio.wtf/v1
kind: StageSet
metadata:
  name: web
  namespace: apps
spec:
  stages:
    - name: web
      sourceRef:
        kind: OCIRepository
        name: web-manifests

Direct: Bucket

Object storage works the same way:

apiVersion: source.toolkit.fluxcd.io/v1
kind: Bucket
metadata:
  name: web-manifests
  namespace: apps
spec:
  interval: 5m
  provider: generic
  bucketName: manifests
  endpoint: minio.storage.svc:9000
  secretRef:
    name: minio-credentials
---
apiVersion: stages.metio.wtf/v1
kind: StageSet
metadata:
  name: web
  namespace: apps
spec:
  stages:
    - name: web
      sourceRef:
        kind: Bucket
        name: web-manifests

Via a renderer (JaaS)

When the source holds Jsonnet rather than plain manifests, render it first. JaaS reads a Flux source with a JsonnetSnippet and publishes the rendered result as an ExternalArtifact, which the stage then consumes:

apiVersion: jaas.metio.wtf/v1
kind: JsonnetSnippet
metadata:
  name: web
  namespace: apps
spec:
  sourceRef:
    kind: GitRepository
    name: web-manifests
    path: ./jsonnet
  entryFile: main.jsonnet
---
apiVersion: stages.metio.wtf/v1
kind: StageSet
metadata:
  name: web
  namespace: apps
spec:
  stages:
    - name: web
      sourceRef:                       # resolve the producer to its ExternalArtifact
        apiVersion: jaas.metio.wtf/v1
        kind: JsonnetSnippet
        name: web

Shared libraries arrive the same way: JOI publishes Jsonnet libraries as single-layer OCI images, surfaced as OCIRepository + JsonnetLibrary pairs a snippet imports:

spec:
  libraries:
    - kind: JsonnetLibrary
      name: k8s-libsonnet
      importPath: k8s          # import 'k8s/...' in your Jsonnet

For small or generated snippets, skip the external source and inline the Jsonnet on the JsonnetSnippet (spec.files). The end-to-end render-and-roll-out flow is in From Jsonnet to a gated rollout.

Symptom

READY=False, REASON=StageFailed. The Message names the stage and the operation that failed (fetch artifact, build, apply, verify, a pre/post action, or connect to target cluster). The run halts at that stage; later stages keep their previous revisions.

Cause

A stage failed during execution. By operation:

• fetch artifact — the artifact URL was unreachable, or its bytes failed digest verification.
• build — kustomize build or post-build substitution failed (a missing substituteFrom source, an invalid patch, a malformed manifest).
• apply — the server-side apply was rejected: an immutable-field conflict, or an RBAC denial under the impersonated serviceAccountName.
• verify — applied objects did not become Ready within the stage timeout (kstatus).
• pre/post action — a patch/http/wait/job/delete/apply action failed or timed out.
• connect to target cluster — a spec.kubeConfig Secret was missing, unparseable, or used the unsupported cloud-provider configMapRef.

Diagnosis

kubectl describe stageset <name> -n <namespace>     # Message: which stage + operation
kubectl -n stageset-system logs deploy/stageset-controller --tail=200

# For apply/verify failures, inspect what the stage tried to apply:
kubectl get stageinventory -n <namespace> \
  -l stages.metio.wtf/stage-set=<name>,stages.metio.wtf/stage=<stage>

Remediation

Match the operation in the Message:

• fetch / digest — confirm the producer republished cleanly; a digest mismatch means the artifact changed mid-flight or is corrupt.
• build — validate the manifests/patches locally; ensure every substituteFrom ConfigMap/Secret exists.
• apply RBAC — grant the impersonated serviceAccountName (or the controller) the verbs it was denied; the Message names the resource.
• apply immutable conflict — set a per-stage conflictPolicy (or force: true, its blunt Recreate-everything form) so the controller deletes and recreates the conflicting object; for objects holding data (PersistentVolumeClaim/PersistentVolume) a Recreate rule additionally requires allowDataLoss: true. Alternatively, use content-hash-suffixed names so a change is a new object rather than a mutation.
• verify timeout — raise the stage timeout, or fix why the workload is not becoming Ready.
• action — read the action’s error; for http, confirm the host is in --allowed-action-hosts.

Retries re-run the same pinned snapshot idempotently — actions already recorded in the stage’s ledger do not re-fire. See stages and sources for how a stage resolves and applies.

apiVersion: stages.metio.wtf/v1
kind: StageInventory

A StageInventory records the set of objects a single stage has applied, so the controller can prune precisely and tear stages down in reverse order. You do not author these — the controller creates, updates, and deletes them. They are documented here so you can read them when debugging and back them up.

One stage may be backed by several StageInventory shards once it exceeds --inventory-shard-cap entries (default 5000). Shard 0 doubles as the ApplySet (KEP-3659) parent object for the stage.

spec

A StageInventory as the controller writes it (read-only — never hand-author one):

apiVersion: stages.metio.wtf/v1
kind: StageInventory
metadata:
  name: payments-application-0          # <stageset>-<stage>-<shard>
  namespace: payments
  labels:
    stages.metio.wtf/stage-set: payments
    stages.metio.wtf/stage: application
    stages.metio.wtf/shard: "0"
spec:
  stagePosition: 1                       # the stage's index in spec.stages
  entries:                               # identifiers only — never object contents
    - id: payments_web_apps_Deployment   # namespace_name_group_kind
      v: apps/v1                         # the applied API version
    - id: payments_web__Service          # empty group → core/v1
      v: v1

The inventory is stored in spec (not status) on purpose: backup tooling that restores spec preserves the prune history, so a restored controller does not orphan or wrongly prune objects.

Well-known labels and annotations

The controller stamps these onto inventories and managed objects: