Performance Tuning (Deep Workflow)
Performance work is measurement-driven. Profile before optimizing; verify after changes; guard against regressions with benchmarks or production metrics.
When to Offer This Workflow
Trigger conditions:
- High CPU, memory, p99 latency, GC pauses
- Cost reduction via efficiency
- Premature optimization requests—need evidence first
Initial offer:
Use six stages: (1) frame goals & SLOs, (2) measure baseline, (3) profile & hypothesize, (4) implement changes, (5) verify & compare, (6) prevent regression). Confirm language/runtime and environment (prod-like data volume).
Stage 1: Frame Goals & SLOs
Goal: Numeric targets: p95 latency, throughput, max memory—not “faster.”
Questions
- Which workloads matter most (batch vs interactive)?
- Correctness constraints (approximation allowed or not)?
- Cost budget for hardware vs engineering time?
Exit condition: One-page success criteria and out-of-scope areas.
Stage 2: Measure Baseline
Goal: Reproducible benchmark or RUM segment—same inputs, same conditions.
Practices
- Warm caches when prod is always warm
- Statistical repeat (multiple runs, discard outliers methodology)
Exit condition: Baseline numbers + environment fingerprint (versions, flags).
Stage 3: Profile & Hypothesize
Goal: Find dominant cost: CPU bound, I/O bound, lock contention, allocation rate.
Tools (examples)
- CPU flame graphs; async wait profiling
- Alloc profiling for GC pressure
- DB query plans and lock waits
Exit condition: Hypothesis tied to evidence (e.g., “40% time in JSON parse”).
Stage 4: Implement Changes
Goal: Smallest change that addresses the hotspot; avoid clever without proof.
Levers
- Algorithm / data structure
- Caching with invalidation discipline
- Batching I/O; connection pooling
- Parallelism where safe—watch locks
Stage 5: Verify & Compare
Goal: A/B or before/after with same workload; watch tail latency not only mean.
Production
- Canary with error rate and latency gates
Stage 6: Prevent Regression
Goal: Micro-benchmarks in CI (optional), budgets, or synthetic checks.
Final Review Checklist
- Goals and baseline documented
- Root cause supported by profiler/trace evidence
- Change scoped; trade-offs explicit
- Verification on realistic load
- Regression guard where feasible
Tips for Effective Guidance
- Little’s Law intuition: queues blow latency—often fix concurrency before micro-opts.
- Avoid optimizing cold paths first.
- GC languages: allocation rate often is the enemy.
Handling Deviations
- Embedded / mobile: battery and thermal constraints matter too.
- Distributed systems: local opt may hurt system (see load-testing).