Kubernetes Incident Response

Runbooks and diagnostic workflows for common Kubernetes incidents.

Requires the cilium, diagnostics, istio, security toolsets: start kubectl-mcp-server with --toolsets default,cilium,diagnostics,istio,security (or MCP_TOOLSETS=default,cilium,diagnostics,istio,security).

When to Apply

Use this skill when:

• User mentions: "incident", "outage", "emergency", "down", "not working"
• Operations: emergency response, production issues, service degradation
• Keywords: "urgent", "broken", "fix", "restore", "recover"

Priority Rules

Quick Reference

Incident Triage

Quick Health Check

kubectl_get(resource_type="nodes")
kubectl_get(resource_type="pods", namespace="kube-system")
kubectl_get(resource_type="events", namespace)

Severity Assessment

Runbook: Pod Failures

CrashLoopBackOff

kubectl_logs(name, namespace, previous=True)
kubectl_describe(name, namespace)
kubectl_get(resource_type="events", namespace, field_selector="involvedObject.name=<pod>")
kubectl_top(name, namespace)

Common Causes:

• OOMKilled → Increase memory limits
• Exit code 1 → Application error in logs
• Exit code 137 → Killed by OOM or SIGKILL
• Exit code 143 → Graceful SIGTERM

ImagePullBackOff

kubectl_describe(name, namespace)
kubectl_get(resource_type="secrets", namespace)

Pending Pod

kubectl_describe(name, namespace)
kubectl_get(resource_type="nodes")
kubectl_get(resource_type="events", namespace)

Runbook: Node Issues

Node NotReady

kubectl_describe(name)
kubectl_get(resource_type="events", namespace="", field_selector="involvedObject.name=<node>")
node_logs(name, "kubelet")

Node DiskPressure

kubectl_describe(name)
kubectl_get(resource_type="pods", field_selector="spec.nodeName=<node>")

Runbook: Network Issues

Service Not Accessible

kubectl_get(resource_type="services", namespace)
kubectl_get(resource_type="endpoints", namespace)
kubectl_get(resource_type="pods", namespace, label_selector="<service-selector>")
analyze_network_policies(namespace)

DNS Resolution Failures

kubectl_get(resource_type="pods", namespace="kube-system", label_selector="k8s-app=kube-dns")
kubectl_logs("coredns-xxx", "kube-system")

With Cilium

cilium_get_status()
cilium_list_endpoints(namespace)
get_hubble_flows(namespace)

With Istio

istio_analyze(namespace)
istio_proxy_status()

Runbook: Storage Issues

PVC Pending

kubectl_describe(name, namespace)
kubectl_get(resource_type="storageclasses")
kubectl_get(resource_type="events", namespace)

Pod Stuck in ContainerCreating

kubectl_describe(name, namespace)
kubectl_get(resource_type="pvc", namespace)
kubectl_get(resource_type="events", namespace)

Runbook: Control Plane Issues

API Server Unavailable

kubectl_get(resource_type="pods", namespace="kube-system", label_selector="component=kube-apiserver")
kubectl_get(resource_type="events", namespace="kube-system")

etcd Issues

kubectl_get(resource_type="pods", namespace="kube-system", label_selector="component=etcd")
kubectl_logs("etcd-xxx", "kube-system")

Emergency Actions

Delete a Stuck Pod

delete_resource(resource_type="pod", name=name, namespace=namespace)

Rollback Deployment

kubectl_rollout(action="undo", resource_type="deployment", name=name, namespace=namespace)

Helm Rollback

helm_rollback(name, namespace, revision=1)

Diagnostic Collection Script

For comprehensive incident diagnostics, see scripts/collect-diagnostics.py.

Multi-Cluster Incident Response

Check all clusters:

for context in ["prod-1", "prod-2", "staging"]:
    kubectl_get(resource_type="nodes", context=context)
    kubectl_get(resource_type="pods", namespace="kube-system", context=context)
    kubectl_get(resource_type="events", namespace="kube-system", context=context)

Post-Incident

Document Timeline

1. When did the incident start?
2. What was the impact?
3. What was the root cause?
4. What fixed it?

Prevent Recurrence

• Add monitoring/alerting
• Improve resource limits
• Add readiness probes
• Document runbook

Related Skills

• k8s-troubleshoot - Detailed debugging
• k8s-security - Security incidents