Symptom

READY=False, REASON=StageFailed. The Message names the stage and the operation that failed (fetch artifact, build, apply, verify, a pre/post action, or connect to target cluster). The run halts at that stage; later stages keep their previous revisions.

Cause

A stage failed during execution. By operation:

• fetch artifact — the artifact URL was unreachable, or its bytes failed digest verification. Transient fetch failures (network, 5xx, source not ready) back off and retry. Terminal ones — a digest mismatch, an oversized tarball, or a URL the SSRF guard rejects — stop requeuing, since the same fetch would fail the same way; the next genuine watch event or interval tick re-runs.
• apply RBAC / missing CRD — a Forbidden apply, or a manifest whose kind the apiserver doesn’t know, is reported as RBACDenied (terminal), not StageFailed.
• build — kustomize build or post-build substitution failed (a missing substituteFrom source, an invalid patch, a malformed manifest).
• apply — the server-side apply was rejected: an immutable-field conflict, or an RBAC denial under the impersonated serviceAccountName.
• verify — an applied object reached a terminal state (kstatus Failed), which aborts the wait immediately. A stage whose objects were still progressing when the timeout elapsed is reported as StageProgressing instead, and converges on its own.
• pre/post action — a patch/http/wait/job/delete/apply action failed or timed out.
• connect to target cluster — a spec.kubeConfig Secret was missing, unparseable, or used the unsupported cloud-provider configMapRef.

Diagnosis

kubectl --namespace <namespace> describe stageset <name>     # Message: which stage + operation
kubectl --namespace stageset-system logs deploy/stageset-controller --tail=200

# For apply/verify failures, inspect what the stage tried to apply:
kubectl --namespace <namespace> get stageinventory \
  --selector stages.metio.wtf/stage-set=<name>,stages.metio.wtf/stage=<stage>

Remediation

Match the operation in the Message:

• fetch / digest — confirm the producer republished cleanly; a digest mismatch means the artifact changed mid-flight or is corrupt.
• build — validate the manifests/patches locally; ensure every substituteFrom ConfigMap/Secret exists.
• apply RBAC — grant the impersonated serviceAccountName (or the controller) the verbs it was denied; the Message names the resource.
• apply immutable conflict — set a per-stage conflictPolicy (or force: true, its blunt Recreate-everything form) so the controller deletes and recreates the conflicting object; for objects holding data (PersistentVolumeClaim/PersistentVolume) a Recreate rule additionally requires allowDataLoss: true. Alternatively, use content-hash-suffixed names so a change is a new object rather than a mutation.
• verify — an object reached a terminal failure; fix why, rather than raising the timeout. (A timeout on a still-progressing object is StageProgressing , where raising the timeout is the answer.)
• action — read the action’s error; for http, confirm the host is in --allowed-action-hosts.

An apply that failed with Unauthorized

Unauthorized is authentication, not authorization, so it is not an RBAC problem — checking the tenant ServiceAccount’s Role will send you down the wrong path. The controller applies as spec.serviceAccountName using a short-lived TokenRequest token, and that token is bound to the ServiceAccount’s UID. Deleting and recreating the ServiceAccount — which happens to every ServiceAccount in a namespace that is torn down and rebuilt — gives it the same name and a new UID, so a token minted before the rebuild authenticates as an object the apiserver no longer knows.

The controller handles this: a 401 evicts the cached credential, mints a fresh one, and retries the call once. A message that names the ServiceAccount and says the fresh token was refused too means the retry did not help, so look at the ServiceAccount itself:

kubectl --namespace <namespace> get serviceaccount <sa-name>

A missing ServiceAccount, or a namespace in Terminating, is the answer. Recreate the ServiceAccount (or let the namespace finish terminating) and the next reconcile applies normally.

Retries re-run the same pinned snapshot idempotently — actions already recorded in the stage’s ledger do not re-fire. See stages and sources for how a stage resolves and applies.