Observability & SLOs (Deep Workflow)
SLOs connect engineering work to user-perceived reliability. SLIs must be measurable from systems but grounded in user journeys.
When to Offer This Workflow
Trigger conditions:
- Defining 99.9% without defining for what
- Too many pages or none; need error budget discipline
- Product wants features while stability degrades
Initial offer:
Use six stages: (1) pick user journeys, (2) define SLIs, (3) set SLO targets & windows, (4) error budget policy, (5) alerting on budget burn, (6) review & iterate). Confirm metric stack and dependency SLOs from vendors.
Stage 1: User Journeys
Goal: Critical paths that matter if broken—checkout, login, API sync, not “CPU low”.
Output
3–10 journeys ranked by business impact and frequency.
Exit condition: One paragraph per journey: user intent + failure symptom.
Stage 2: Define SLIs
Goal: Ratio of good events over total over a window—implementation explicit.
Examples
- Availability: successful requests / valid requests (define “valid”)
- Latency: proportion of requests faster than T ms
Good SLIs
- Objective, low-cardinality enough to measure reliably
Exit condition: SLI formula + data source (metrics, logs, probes).
Stage 3: SLO Targets & Windows
Goal: Target (e.g., 99.9% monthly) implies allowed bad minutes—make it explicit.
Practices
- Rolling 30d common; align with release cadence
- Tier services: not everything needs same SLO
Exit condition: Published table: journey → SLI → target → window.
Stage 4: Error Budget Policy
Goal: What we do when budget is healthy vs exhausted.
Policy ideas
- Budget healthy → ship features; low → freeze risky changes, focus on reliability
- Escalation when budget burns fast (multi-window alerts)
Exit condition: Written policy with product sign-off.
Stage 5: Alerting on Burn
Goal: Page on budget burn rate, not every blip—multi-window multi-burn-rate pattern when using Google-style SLO alerting.
Practices
- Fast burn = page soon; slow burn = ticket/track
Exit condition: Alert rules linked to runbooks.
Stage 6: Review & Iterate
Goal: SLOs drift with architecture—quarterly review; adjust targets with data.
Final Review Checklist
- Journeys and SLIs tied to real user pain
- Targets realistic vs dependencies and cost
- Error budget policy agreed with product
- Alerts on burn, not noisy symptom spam
- Review cadence scheduled
Tips for Effective Guidance
- Translate 99.9% to minutes/month of allowed badness.
- SLA (contract) vs SLO (internal)—don’t confuse.
- Dependency SLO caps what you can promise—surface that early.
Handling Deviations
- No metrics yet: start with proxy SLI (synthetic probes) and improve instrumentation.
- Batch systems: event processing lag as SLI instead of HTTP.