TeardownForced
Symptom
A StageSet that was deleted lingered in Terminating, then disappeared. A Warning Event records the force-drop:
kubectl --namespace <ns> get events --field-selector reason=TeardownForced
...
Warning TeardownForced TeardownForced after 1h0m0s of failing teardown (delete stage "deploy" objects) —
finalizer dropped; the target cluster may carry orphaned objects an operator must remove by hand. Last error: ...
The stageset_teardown_force_drop_total{namespace,name,reason} counter increments at the same time.
Cause
On deletion the controller tears the StageSet’s applied objects down in reverse stage order, then drops the finalizer. While a teardown step keeps failing the finalizer is held and the delete retries — so a transient target-cluster outage heals on its own.
But a permanently-unreachable target — a deleted spec.kubeConfig Secret, revoked tenant RBAC, or a decommissioned remote cluster — would wedge the StageSet in Terminating forever and block namespace teardown. Two things end that wait, and the counter’s reason label says which:
reason | Meaning |
|---|---|
timed_out | The deletion has been pending longer than --max-teardown-wait (default 1h). The target may merely be slow or intermittently unreachable, so it gets the full window first. |
permanent | The teardown failed with an error no retry clears — Forbidden on the delete, or a kind the target no longer serves. An operator has to act, and the StageSet already on its way out is not what they will act on. |
unauthorized | The tenant credential was refused even after the controller evicted it and minted a fresh one, so the ServiceAccount the objects would be deleted as is gone. The usual reason is that its own namespace is terminating — and holding the finalizer for that pins the namespace behind objects it is collecting anyway. |
The last two drop immediately rather than waiting the bound out: the orphaned-object cost is identical either way, and an hour in Terminating first buys nothing. Whatever objects the failing stage could not delete are left orphaned.
Diagnosis
Identify which target and stage failed from the Event message (Last error and the operation, e.g. delete stage "deploy" objects). Then confirm what was left behind:
# Objects the controller applies carry owner labels keyed by the StageSet name.
kubectl --context <target-context> get all,configmap,secret --all-namespaces \
-l stages.metio.wtf/name=<stageset-name>
If the target was a remote cluster (spec.kubeConfig), check that the kubeconfig Secret and the cloud identity it referenced still exist.
Remediation
- Restore access to the target if it is meant to keep running (re-create the kubeConfig Secret, re-grant the tenant SA’s RBAC), then delete the orphaned objects with the label selector above.
- If the target is genuinely gone, the orphaned objects went with it — nothing to clean up.
To make the controller wait longer before force-dropping (for example, to ride out a planned multi-hour target outage), raise --max-teardown-wait. Setting it very high re-introduces the original wedge risk, so prefer fixing the target over disabling the escape hatch. It bounds the timed_out reason only — a permanent or unauthorized cause drops regardless, since waiting cannot change the outcome.