Symptom

A stageset-controller pod is NotReady; the StageSetControllerPodDown alert fires. While no replica is Ready, StageSets are not reconciled and the Kubernetes admission webhook may reject StageSet writes (failurePolicy: Fail).

Cause

• the manager cannot start, so /readyz never goes green while the pod keeps running — denied egress to the apiserver, an incomplete ClusterRole, missing CRDs, a webhook certificate that has not been issued. GET /manager on the probe port names the reason; see manager unavailable ,
• a crash-looping container (bad config flag, panic),
• the node draining or out of resources,
• the leader-election lease unobtainable.

Diagnosis

kubectl --namespace stageset-system get pods --selector app.kubernetes.io/name=stageset-controller
kubectl --namespace stageset-system describe pod <pod>
kubectl --namespace stageset-system logs <pod> --previous --tail=200

Look for flag-parse errors at startup, RBAC Forbidden on the controller’s own ServiceAccount, or OOMKills.

Remediation

• Read GET /manager on the probe port first: a pod that is Running and 0/1 with no restarts is a manager that cannot start, and that page names the cause and the fix. The pod recovers on its own once it is cleared.
• Fix the surfaced cause (correct the flag/values, grant the missing controller RBAC, raise resource limits).
• Run more than one replica with leader election so a single pod failure doesn’t stop reconciliation — see production .
• If admission is blocking writes during the outage and you must unblock urgently, scope or relax the webhook failurePolicy, then restore it once the controller is healthy.

See operations for the full alert set and its thresholds.