Error-budget freeze see history edit this page

Talks about: , , , and

An error-budget freeze holds new-revision rollouts while a service is out of its SLO error budget, and resumes them on its own once the budget recovers — the Google SRE error-budget policy: when the budget is spent, stop shipping feature changes until reliability is back. The controller does no SLO math; it reads one number (the remaining budget) from a metric source you already run and compares it to a threshold.

Freeze on a Prometheus query

Point spec.errorBudget at an instant query that returns the remaining budget as a scalar — every SLO tool exposes this (Sloth records slo:period_error_budget_remaining:ratio directly; Pyrra, Grafana SLO, and Nobl9’s Prometheus API yield it in one line):

apiVersion: stages.metio.wtf/v1
kind: StageSet
metadata:
  name: checkout
  namespace: apps
spec:
  serviceAccountName: deployer
  errorBudget:
    source:
      prometheus:
        address: http://prometheus.monitoring:9090
        query: slo:period_error_budget_remaining:ratio{sloth_service="checkout",sloth_slo="availability"}
    freezeThreshold: "0"      # freeze when remaining < this; "0" = only when overspent
    resumeThreshold: "0.05"   # resume only once it recovers to here (hysteresis)
    interval: 5m              # re-check cadence while frozen; defaults to spec.interval
  stages:
    - name: prod
      sourceRef:
        name: checkout-prod

While the remaining budget is below freezeThreshold, the StageSet reports Ready=False with reason BudgetExhausted (an already-deployed StageSet stays Ready=True and records the freeze on status.budgetFreeze). The current revision keeps having its drift corrected — a frozen service still gets its declared state enforced — and only new-revision rollouts wait. The controller re-checks every interval and advances on its own once the budget reaches resumeThreshold.

freezeThreshold is required and has no default. resumeThreshold defaults to freezeThreshold (no hysteresis); set it higher to stop a budget hovering at the line from flapping the freeze. interval defaults to the StageSet’s reconcile interval.

It composes with update windows

spec.errorBudget and spec.updateWindows are combined under a logical AND: a new revision rolls out only if the update window is open and the budget is healthy. A closed window holds the rollout even when the budget is fine (and the budget source isn’t even queried), and an exhausted budget holds it even inside an open window. Use both to deploy only during a maintenance window and only while in budget.

Authenticating the query

When Prometheus needs a bearer token, reference a Secret in the StageSet’s namespace with the token under the token key:

  errorBudget:
    source:
      prometheus:
        address: https://prometheus.monitoring:9090
        query: slo:period_error_budget_remaining:ratio{sloth_service="checkout"}
        secretRef:
          name: prometheus-auth

A metric source is an outbound call to an address the StageSet names, so it carries the same two guards an HTTP action does. Every resolved address is pinned through the SSRF guard: loopback, link-local, cloud-metadata, multicast, and unspecified addresses are refused, while in-cluster private addresses (where Prometheus usually lives) are allowed. And when the controller runs with --allowed-action-hosts, the source’s host must match one of its patterns — leave the flag unset to reach any host the SSRF guard permits.

The secretRef bearer token is read as the StageSet’s spec.serviceAccountName, in the StageSet’s own namespace, so grant that ServiceAccount get on the Secret you reference. The token reaches only endpoints your ServiceAccount’s own credentials could already reach.

Reading from a SaaS SLO API (webhook)

When the budget lives in a SaaS SLO platform with no Prometheus endpoint (Nobl9, Grafana Cloud), use a webhook source instead: the controller GETs a JSON document and extracts the scalar with a kubectl-style JSONPath.

  errorBudget:
    source:
      webhook:
        url: https://nobl9.example/api/v2/slos/shop/checkout
        jsonPath: "{.objectives[0].errorBudgetRemaining}"
        secretRef:
          name: nobl9-token        # optional bearer token, under the "token" key
    freezeThreshold: "0"

A source is exactly one of prometheus or webhook. The jsonPath must resolve to a single numeric (or numeric-string) value; anything else (no match, multiple matches, an object, NaN) is treated as an unreadable source and routed through onSourceError. The webhook URL is dialed through the same SSRF guard.

Freezing a single stage

spec.errorBudget freezes the whole rollout. To freeze only one stage on its own SLO — e.g. hold prod while prod’s budget is spent, but let staging keep rolling — put an errorBudget on that stage:

  stages:
    - name: staging
      sourceRef:
        name: web-staging
    - name: prod
      sourceRef:
        name: web-prod
      errorBudget:
        source:
          prometheus:
            address: http://prometheus.monitoring:9090
            query: slo:period_error_budget_remaining:ratio{sloth_service="checkout-prod"}
        freezeThreshold: "0.1"

A per-stage budget gates entry to the stage: while it is exhausted, a new revision is held from rolling into that stage (earlier stages keep rolling, and the stage’s currently-applied revision keeps having its drift corrected). It takes the same fields as the rollout-wide budget (freezeThreshold/resumeThreshold hysteresis, interval, onSourceError, dryRun) and the same budget-override break-glass clears both. This complements a promotion analysis , which gates exit (advancing past a stage once it has applied).

When the source is unreachable

onSourceError decides what happens when the query can’t be read (Prometheus down, a non-2xx, an empty result, NaN):

A misconfigured query that always returns NaN would silently fail open, so set dryRun: true to prove a freeze rule fires before it gates — it records what would freeze on status.budgetFreeze and the metrics, without holding anything.

Shipping a reliability fix while frozen

The error-budget policy explicitly exempts reliability and security fixes. Break the glass to apply a held rollout once, without disabling the gate:

stagesetctl reconcile checkout --namespace apps --budget-override

What it observes

The freeze reads remaining budget, not burn rate — a fast-burning but not-yet-exhausted service still rolls out. Multi-window burn-rate alerting is the SLO tool’s job. To gate on observed behavior per stage (error rate, latency, burn rate) rather than rollout-wide, use a promotion analysis , which shares this same metric source contract.