Metrics

The controller registers custom metrics on the controller-runtime registry, served on --metrics-bind-address (:8080) alongside the standard controller_runtime_* and workqueue_* series. Enable scraping with the chart’s opt-in ServiceMonitor (metrics.serviceMonitor.enabled):

# values.yaml
metrics:
  serviceMonitor:
    enabled: true        # needs the Prometheus operator CRDs

Degraded manager

The controller manager is supervised independently of the process that hosts it. When it cannot be built or started — an apiserver the pod cannot reach, a ClusterRole missing a verb, a CRD not yet installed, a webhook certificate not issued yet — the process keeps running and retries the manager with exponential backoff (1s, doubling, capped at 5m) for as long as the pod lives. The manager comes up on its own once the cause is cleared, in the same process and without a restart.

That matters because the probe and metrics endpoints are bound by the binary rather than by the manager, so they keep answering while the manager is down and the cause stays legible:

• GET /healthz stays an unconditional 200, so the kubelet never restarts the pod over a degraded manager.
• GET /readyz reports not-ready, so the pod stays out of its Services. Nothing it serves works without the manager — the webhook, the stage-gate endpoint and the MCP server are all manager-scoped — so withdrawing it is the honest answer.
• GET /manager carries the reason, the time the reading last changed, and the number of manager starts.
• stageset_manager_available is 0 while the manager is down; stageset_manager_start_failures_total separates one long outage from a manager that keeps dying.
• The log carries one manager unavailable, restarting line per fresh cause and a still unavailable line per retry, each naming the reason. The backoff paces them, so a long outage thins out to one line every five minutes.

A manager that lasted at least a minute before failing gets a fresh backoff; one that fails quickly keeps the growing delay, so a controller that cannot stay up does not hammer the apiserver.

Diagnosis is in the manager-unavailable runbook .

Alerts

The chart ships an opt-in PrometheusRule with a starter alert set, gated on metrics.prometheusRule.enabled (requires the Prometheus operator CRDs). It covers the custom stageset_* metrics plus controller-runtime signals:

Every threshold is a knob under metrics.prometheusRule.thresholds, and extraAlertLabels is merged onto every rendered alert so all stageset alerts can route through one Alertmanager receiver. Each alert carries a runbook_url annotation pointing at the matching runbook page on this site; the URL prefix is fixed to this site, and the reconcile-errors alert templates the URL on $labels.reason. Append your own rules under metrics.prometheusRule.extraRules, and silence a built-in alert by raising its threshold rather than forking the chart. When StageSetControllerWorkqueueDepthHigh or StageSetReconcileLatencyHigh fires persistently, the controller is at capacity — see Scale and capacity for which lever to pull.

Events

The controller emits Kubernetes Events on every Ready-condition transition, so kubectl describe stageset <name> and Flux ’s notification-controller (via an Alert targeting kind: StageSet) both surface what happened. Normal events include Succeeded, UpdateDeferred, MigrationStarted, and MigrationCompleted; warnings include StageFailed, DriftCorrected, RolledBack, MigrationFailed, OnFailureAction, RollbackStoreFailed, and TeardownForced (a deleting StageSet’s finalizer force-dropped after --max-teardown-wait — see TeardownForced ).

Runbooks

Every actionable Ready-condition reason has a runbook covering the symptom, cause, diagnosis, and remediation. The controller appends a (runbook: https://stageset.projects.metio.wtf/runbooks/<reason>/) link to every actionable Ready message (the reason lower-cased into a path segment), so a kubectl describe links straight to the fix. Healthy reasons (Succeeded, Suspended) get no link.

For example, a StageFailed StageSet shows:

Message:  stage "application" failed: … (runbook: https://stageset.projects.metio.wtf/runbooks/stagefailed/)

Forcing a reconcile

The controller reconciles on its spec.interval, on source changes, and on demand. To trigger an out-of-band run, stamp the standard annotation — which is what flux reconcile and stagesetctl reconcile do for you:

kubectl annotate stageset my-app \
  reconcile.fluxcd.io/requestedAt="$(date -u +%FT%TZ)" --overwrite

The handled token is recorded in status.lastHandledReconcileAt.

Drift correction

On a steady-state reconcile the controller re-asserts the desired state, healing out-of-band changes to managed objects. Each correction emits a DriftCorrected event and increments stageset_drift_corrected_total. Tighten the cadence with spec.driftDetectionInterval when you need faster healing than spec.interval.