Symptom

workqueue_depth{controller="stageset"} stays high; StageSets reconcile slowly or lag behind their spec.interval. The StageSetControllerWorkqueueDepthHigh alert fires (see operations for the alert set and its thresholds).

Cause

The controller is enqueuing reconcile requests faster than it completes them. Common causes:

• apiserver slowness — applies, dry-runs, and status writes all block on the apiserver (or the impersonated tenant’s authorization).
• slow sources — a stage waiting on a large artifact fetch or a source that is slow to become Ready holds a worker.
• a stuck stage — an action with a long timeout (a wait/http/job that never completes) pins a worker for the whole timeout. So does a stage’s verify wait: a workload that never reports ready holds its worker until the stage timeout elapses (15 minutes by default), and every event for that StageSet — including its own deletion — waits behind it, since a key reconciles one at a time.
• too few workers for the StageSet count — many StageSets reconciling on short intervals.

Diagnosis

# which StageSets are churning?
kubectl get stagesets --all-namespaces --sort-by=.status.observedGeneration
# controller logs for slow operations / retries
kubectl --namespace stageset-system logs deploy/stageset-controller --tail=200

Correlate with controller_runtime_reconcile_time_seconds (see reconcile latency ) and apiserver latency.

workqueue_longest_running_processor_seconds{name="stageset"} answers the question the log cannot: a value climbing towards a stage timeout means a worker is parked inside a reconcile, not that the controller has stopped. Zero, with a non-empty workqueue_depth, points elsewhere.

Remediation

• Lengthen spec.interval on high-churn StageSets that don’t need fast reconciliation.
• Cap long-running actions with a tighter timeout so a stuck action releases its worker.
• Adding replicas does not help: leader election means only the lease holder reconciles, so a second replica is failover, not added throughput (production ). The controller has no reconcile-concurrency flag — the levers are reducing load (longer intervals, fewer StageSets, fewer objects per stage) and removing the slow operations below.
• Investigate apiserver / tenant-authorization latency if reconciles are uniformly slow.