Error-budget freeze

An error-budget freeze holds new-revision rollouts while a service is out of its SLO error budget, and resumes them on its own once the budget recovers — the Google SRE error-budget policy: when the budget is spent, stop shipping feature changes until reliability is back. The controller does no SLO math; it reads one number (the remaining budget) from a metric source you already run and compares it to a threshold.
Freeze on a Prometheus query
Point spec.errorBudget at an instant query that returns the remaining budget as
a scalar — every SLO tool exposes this (Sloth records
slo:period_error_budget_remaining:ratio directly; Pyrra, Grafana SLO, and
Nobl9’s Prometheus API yield it in one line):
apiVersion: stages.metio.wtf/v1
kind: StageSet
metadata:
name: checkout
namespace: apps
spec:
serviceAccountName: deployer
errorBudget:
source:
prometheus:
address: http://prometheus.monitoring:9090
query: slo:period_error_budget_remaining:ratio{sloth_service="checkout",sloth_slo="availability"}
freezeThreshold: "0" # freeze when remaining < this; "0" = only when overspent
resumeThreshold: "0.05" # resume only once it recovers to here (hysteresis)
interval: 5m # re-check cadence while frozen; defaults to spec.interval
stages:
- name: prod
sourceRef:
name: checkout-prod
While the remaining budget is below freezeThreshold, the StageSet reports
Ready=False with reason BudgetExhausted (an already-deployed StageSet stays
Ready=True and records the freeze on status.budgetFreeze). The current
revision keeps having its drift corrected — a frozen service still gets its
declared state enforced — and only new-revision rollouts wait. The controller
re-checks every interval and advances on its own once the budget reaches
resumeThreshold.
freezeThreshold is required and has no default. resumeThreshold defaults to
freezeThreshold (no hysteresis); set it higher to stop a budget hovering at the
line from flapping the freeze. interval defaults to the StageSet’s reconcile
interval.
It composes with update windows
spec.errorBudget and spec.updateWindows
are combined
under a logical AND: a new revision rolls out only if the update window is open and the budget is
healthy. A closed window holds the rollout even when the budget is fine (and the
budget source isn’t even queried), and an exhausted budget holds it even inside
an open window. Use both to deploy only during a maintenance window and only
while in budget.
Authenticating the query
When Prometheus needs a bearer token, reference a Secret in the StageSet’s
namespace with the token under the token key:
errorBudget:
source:
prometheus:
address: https://prometheus.monitoring:9090
query: slo:period_error_budget_remaining:ratio{sloth_service="checkout"}
secretRef:
name: prometheus-auth
A metric source is an outbound call to an address the StageSet names, so it
carries the same two guards an HTTP action does. Every resolved address is
pinned through the SSRF guard: loopback, link-local, cloud-metadata, multicast,
and unspecified addresses are refused, while in-cluster private addresses (where
Prometheus usually lives) are allowed. And when the controller runs with
--allowed-action-hosts, the source’s host must match one of its patterns —
leave the flag unset to reach any host the SSRF guard permits.
The secretRef bearer token is read as the StageSet’s spec.serviceAccountName,
in the StageSet’s own namespace, so grant that ServiceAccount get on the Secret
you reference. The token reaches only endpoints your ServiceAccount’s own
credentials could already reach.
Reading from a SaaS SLO API (webhook)
When the budget lives in a SaaS SLO platform with no Prometheus endpoint (Nobl9,
Grafana Cloud), use a webhook source instead: the controller GETs a JSON
document and extracts the scalar with a kubectl-style JSONPath.
errorBudget:
source:
webhook:
url: https://nobl9.example/api/v2/slos/shop/checkout
jsonPath: "{.objectives[0].errorBudgetRemaining}"
secretRef:
name: nobl9-token # optional bearer token, under the "token" key
freezeThreshold: "0"
A source is exactly one of prometheus or webhook. The jsonPath must resolve
to a single numeric (or numeric-string) value; anything else (no match, multiple
matches, an object, NaN) is treated as an unreadable source and routed through
onSourceError. The webhook URL is dialed through the same SSRF guard.
Freezing a single stage
spec.errorBudget freezes the whole rollout. To freeze only one stage on its own
SLO — e.g. hold prod while prod’s budget is spent, but let staging keep
rolling — put an errorBudget on that stage:
stages:
- name: staging
sourceRef:
name: web-staging
- name: prod
sourceRef:
name: web-prod
errorBudget:
source:
prometheus:
address: http://prometheus.monitoring:9090
query: slo:period_error_budget_remaining:ratio{sloth_service="checkout-prod"}
freezeThreshold: "0.1"
A per-stage budget gates entry to the stage: while it is exhausted, a new
revision is held from rolling into that stage (earlier stages keep rolling, and
the stage’s currently-applied revision keeps having its drift corrected). It takes
the same fields as the rollout-wide budget (freezeThreshold/resumeThreshold
hysteresis, interval, onSourceError, dryRun) and the same
budget-override
break-glass clears
both. This complements a promotion analysis
, which
gates exit (advancing past a stage once it has applied).
When the source is unreachable
onSourceError decides what happens when the query can’t be read (Prometheus
down, a non-2xx, an empty result, NaN):
Allow(default) — proceed. Blocking a rollout-wide freeze would stop every deploy, including the hotfix you need during the very outage that took the source down, so the freeze fails open. The error is still loud: aBudgetSourceUnavailableWarning event and thestageset_metric_source_errors_totalmetric. See the BudgetSourceUnavailable runbook .Hold— block. Set this for a service that must stop deploying when its SLO source is down, accepting that a source outage then blocks its rollouts.
A misconfigured query that always returns NaN would silently fail open, so set
dryRun: true to prove a freeze rule fires before it gates — it records what
would freeze on status.budgetFreeze and the metrics, without holding
anything.
Shipping a reliability fix while frozen
The error-budget policy explicitly exempts reliability and security fixes. Break the glass to apply a held rollout once, without disabling the gate:
stagesetctl reconcile checkout --namespace apps --budget-override
What it observes
The freeze reads remaining budget, not burn rate — a fast-burning but not-yet-exhausted service still rolls out. Multi-window burn-rate alerting is the SLO tool’s job. To gate on observed behavior per stage (error rate, latency, burn rate) rather than rollout-wide, use a promotion analysis , which shares this same metric source contract.