Linux Kernel Crash Debugging

This skill guides you through analyzing Linux kernel crash dumps using the crash utility.

Installation

Install only when the user requests installation. These commands place skill files in the user workspace; they do not authorize services, startup hooks, or background jobs.

Claude Code

claude skill install linux-kernel-crash-debug.skill

OpenClaw

# Method 1: Install via ClawHub
clawhub install linux-kernel-crash-debug

# Method 2: Manual installation
mkdir -p ~/.openclaw/workspace/skills/linux-kernel-crash-debug
cp SKILL.md ~/.openclaw/workspace/skills/linux-kernel-crash-debug/

Quick Start

Language choice: Respond in the user’s preferred language. SKILL_CN.md and Chinese references are optional companion material; translate relevant explanations when requested. Source titles retain their original language for attribution.

Starting a Session

# Analyze a dump file
crash vmlinux vmcore

# Debug a running system
crash vmlinux

# Raw RAM dump
crash vmlinux ddr.bin --ram_start=0x80000000

Core Debugging Workflow

0. Preserve checksums, vmcore-dmesg, build IDs, modules, config, and command line
1. crash> sys              # Validate release/build and panic context
2. crash> log              # Find the FIRST anomaly, not only the last panic
3. crash> bt / bt -a       # Compare panic task with all active CPUs
4. crash> mod              # Confirm faulting module symbols are available
5. crash> struct / kmem    # Test a specific object-lifetime hypothesis
6. Search upstream and verify good/bad kernels before claiming a regression

Read references/evidence-first-workflow.md before deep analysis. It defines the evidence-quality gates, failure-type routing, hypothesis ledger, tool escalation rules, and root-cause report format. Never equate the panic task, fault site, corruption site, and root cause without supporting evidence.

For worked evidence chains, read Cases 13–15 in references/case-studies.md: GRO header-offset reconstruction, InfiniBand error-path double free, and filesystem writeback/reclaim deadlock. Treat a bisect result as a candidate trigger, an NMI frame as interrupted context, and collection overhead as a possible contributor to stalls; verify each before assigning a root cause.

Live-System Safety Contract

Default to offline, read-only analysis. Treat sudo, module load/unload, boot or service configuration, writes under debugfs or /proc/sys, live tracing, and SysRq actions as live-host mutations.

• Do not perform a mutation unless the user explicitly authorizes the exact host and action and confirms a lab or approved maintenance context. If the environment is unknown, stop after read-only checks and provide a runbook.
• Before authorized tracing or detector changes, record the baseline, set a narrow target and time limit, choose a protected output path, and define the cleanup/rollback command. Apply cleanup in the same session and report it.
• Minimize captured arguments and payloads. Trace output and vmcores may expose credentials, paths, keys, and process memory; never upload or share them without explicit approval and an approved sanitization process.
• An agent must never initiate a deliberate panic, reboot, SysRq crash, or kdumpctl test. Explain the prerequisites and hand the final trigger to an authorized human following an approved drill with console access, backups, workload evacuation, and a verified rollback/recovery plan.

🤖 Agent Execution Directives

If you are an AI/Agent using this skill, do not invoke crash interactively as it will block your subshell.

1. Use the bundled wrapper ./scripts/agent-crash.sh for offline regular-file dumps under an unprivileged account. It enforces a narrow command allowlist and truncates displayed output:
   • ./scripts/agent-crash.sh -k vmlinux -c vmcore triage - Runs sys, a high-signal log index, panic/all-CPU backtraces, and module inventory.
   • ./scripts/agent-crash.sh -k vmlinux -c vmcore flow-oom - Top 15 memory checks.
   • ./scripts/agent-crash.sh -k vmlinux -c vmcore flow-deadlock - Pulls UN task stacks.
   • ./scripts/agent-crash.sh -k vmlinux -c vmcore dis-regs <func> <pid> - Assembly regression.
   • ./scripts/agent-crash.sh -k vmlinux -c vmcore check-poison <addr> - Pattern match memory poisons.
2. Restricted primitives: Use ./scripts/agent-crash.sh -k vmlinux -c vmcore read-memory ffff880123456780 64, backtrace <pid>, or disassemble <symbol> when macros are insufficient. run is removed. Both input paths are mandatory; live /proc, /sys, /dev inputs, root execution, and local/home .gdbinit files are rejected. Startup .crashrc files are disabled. No arbitrary crash/GDB commands, shell escapes, pipelines, redirection, command files, or extensions are exposed by this wrapper. Memory reads are limited to 1..256 words. Keep dumps and collected output access-controlled. Use a trusted installed crash executable in an isolated analysis environment; the wrapper is not an OS sandbox or a defense against parser vulnerabilities in malformed dumps/symbol files. If a needed query is unsupported, report the evidence gap and hand a precise offline query to an authorized human rather than bypassing the wrapper.
3. Check references/agentic-heuristics.md for extended expert methodologies.
4. Follow references/evidence-first-workflow.md: report symbol/dump quality, identify the earliest anomaly, keep competing hypotheses, and attach a confidence level plus a disproof test to the conclusion.

Prerequisites

Package Installation

Package installation changes the host and may need administrator privileges. Require explicit authorization for the exact host and packages; prefer already-installed tools. Offline dump analysis runs as an unprivileged account.

Anolis OS / Alibaba Cloud Linux

# Install crash utility
sudo dnf install crash

# Install kernel debuginfo (match your kernel version)
sudo dnf install kernel-debuginfo-$(uname -r)

# Install additional analysis tools
sudo dnf install gdb readelf objdump makedumpfile

# Optional: Install kernel-devel for source code reference
sudo dnf install kernel-devel-$(uname -r)

RHEL / CentOS / Rocky / AlmaLinux

sudo dnf install crash gdb binutils makedumpfile kexec-tools
# Enable the matching debuginfo repository first if needed
sudo dnf debuginfo-install kernel-$(uname -r)

Ubuntu / Debian

sudo apt install crash kdump-tools kexec-tools gdb binutils makedumpfile
# Debian and Ubuntu use different debug-symbol repositories/package suffixes;
# query the exact running-kernel package before installing it.
apt-cache search "linux-image-$(uname -r).*dbg\|linux-image-$(uname -r).*dbgsym"

SLES / openSUSE

sudo zypper install crash kexec-tools makedumpfile
# Optional SUSE kdump UI and matching kernel debuginfo
sudo zypper install yast2-kdump
zypper se -s 'kernel*debug*'

Self-compiled Kernel

# Enable debug symbols in kernel config
make menuconfig  # Enable CONFIG_DEBUG_INFO, CONFIG_DEBUG_INFO_REDUCED=n

# Or set directly
scripts/config --enable CONFIG_DEBUG_INFO
scripts/config --enable CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT

Verify Installation

# Check crash version
crash --version

# Verify debuginfo matches kernel
crash /usr/lib/debug/lib/modules/$(uname -r)/vmlinux /proc/kcore

Core Command Reference

Debugging Analysis

Tasks and Processes

Memory Operations

bt Command Details

The most important debugging command:

crash> bt              # Current task stack
crash> bt -a           # All CPU active tasks
crash> bt -f           # Expand stack frame raw data
crash> bt -F           # Symbolic stack frame data
crash> bt -l           # Show source file and line number
crash> bt -e           # Search for exception frames
crash> bt -v           # Check stack overflow
crash> bt -R <sym>     # Only show stacks referencing symbol
crash> bt <pid>        # Specific process

Context Management

Crash session has a "current context" affecting bt, files, vm commands:

crash> set              # View current context
crash> set <pid>        # Switch to specified PID
crash> set <task_addr>  # Switch to task address
crash> set -p           # Restore to panic task

Session Control (Human Reference)

The following interpreter features are for an authorized human in a trusted offline session. Agents must use the wrapper above; do not invoke GDB passthrough, redirection, or command-file inclusion as a fallback. Review command files before any human use and protect output paths.

# Output control
crash> set scroll off   # Disable pagination
crash> sf               # Alias for scroll off

# Output redirection
crash> foreach bt > bt.all

# GDB passthrough
crash> gdb bt           # Single gdb invocation
crash> set gdb on       # Enter gdb mode
(gdb) info registers
(gdb) set gdb off

# Read commands from file
crash> < commands.txt

ARM64 / x86_64 Quick Reference

Architecture Differences in crash Analysis

For complete ARM64 address parameter derivation, see references/arm64-crash-params.md. For kdump end-to-end setup, see references/kdump-setup-guide.md.

ARM64 Raw/Damaged-Dump Fallback Template

crash_arm64 \
  -m vabits_actual=39 \
  -m phys_offset=0x80000000 \
  -m kimage_voffset=0xffffffc000000000 \
  -m kaslr=0x0 \
  vmlinux vmcore

First try crash vmlinux vmcore. Use explicit values only when VMCOREINFO is absent/damaged or the input is raw RAM. kaslr=0 means KASLR was disabled; never assume that value or reuse another boot's parameters.

Typical Debugging Scenarios

Kernel BUG Location

crash> sys                    # Confirm panic
crash> log | tail -50         # View logs
crash> bt                     # Call stack
crash> bt -f                  # Expand frames for parameters
crash> struct <type> <addr>   # Inspect data structures

Deadlock Analysis

crash> bt -a                  # All CPU call stacks
crash> ps -m | grep UN        # Uninterruptible processes
crash> foreach UN bt          # View waiting reasons
crash> struct mutex <addr>    # Inspect lock state

Memory Issues

crash> kmem -i                # Memory statistics
crash> kmem -S <cache>        # Inspect slab
crash> vm <pid>               # Process memory mapping
crash> search -k <pattern>    # Search memory

Stack Overflow

crash> bt -v                  # Check stack overflow
crash> bt -r                  # Raw stack data

Advanced Techniques

Recovering Lock Pointers and Mutex Owners (ARM64)

Sources: mutex lock pointer and rwsem lock derivation, Kernel Panic Lab.

When a task sleeps in mutex_lock() or a rwsem slow path, trace the first ARM64 argument (x0) at the call site:

# Path A: a global lock is constructed directly
    adrp x0, 0xffffffc00ac1e000
    add  x0, x0, #0x7f0         # lock = page + offset
    bl   mutex_lock

# Path B: the caller passes a callee-saved register
    mov  x0, x19                # lock pointer is the saved x19 value
    bl   mutex_lock
# Find the exact "add x29, sp, #N" and "stp/str ..., x19, [sp,#M]",
# derive SP from the frame pointer, then read the x19 stack slot with rd.

crash> struct mutex <lock_addr> -x
# mutex.owner packs flags into its low 3 bits on common kernels:
# owner_task = owner.counter & ~0x7
crash> struct task_struct <owner_task>
crash> bt <owner_pid>

stp x20, x19, [sp,#32] saves x20 at sp+32 and x19 at sp+40. Never reuse example offsets blindly: derive them from the vmcore's matching vmlinux. Verify the mutex layout and owner flag definitions against the analyzed kernel.

For exact FP/SP arithmetic, stp slot ordering, owner masking, and failure checks, read references/arm64-lock-analysis.md. For a complete rwsem example, read Case 11 in references/case-studies.md.

x86_64 equivalent: Use RBP chain with bt -f. Note that with -fomit-frame-pointer, this technique may fail; in that case use bt -F or look for explicit stack frames.

Memory Leak Diagnostic (Three-Layer Check)

Source: Kernel panic 实验室 - Kernel driver 内存泄露问题排查指南

Three independent paths to diagnose memory leaks:

The first path is read-only. Enabling page_owner or writing kmemleak controls changes a live kernel and must follow the Live-System Safety Contract above. Preserve the current output before clearing detector state.

# === Layer 1: /proc 三件套 (read from running system or captured info) ===
# MemAvailable 持续下降 + SUnreclaim 持续增加 → slab 内存泄露
cat /proc/meminfo
cat /proc/slabinfo
cat /proc/buddyinfo

# === Layer 2: SLAB-specific (slub_debug) ===
# In bootargs: slub_debug=u,kmalloc-512
# Then read:
cat /sys/kernel/debug/slab/kmalloc-512/alloc_traces
cat /sys/kernel/debug/slab/kmalloc-512/free_traces

# === Layer 3: >8K allocations (page_owner) ===
# SUnreclaim rises but slabinfo flat → kmalloc > 8K uses alloc_pages directly
# Enable CONFIG_PAGE_OWNER + boot with page_owner=on
# If page_owner is not already enabled, use an approved maintenance runbook;
# do not enable it as part of automated triage.
# Periodic dumps, then diff:
./page_owner_sort --cull name,ator,stacktrace page_owner_begin.txt > begin.txt
./page_owner_sort --cull name,ator,stacktrace page_owner_end.txt   > end.txt
# Compare begin.txt vs end.txt - rising stacks are leaks

# === Alternative: kmemleak ===
# CONFIG_DEBUG_KMEMLEAK + kmemleak=on bootarg
echo scan > /sys/kernel/debug/kmemleak
cat /sys/kernel/debug/kmemleak

Chained Queries

crash> bt -f                  # Get pointers
crash> struct file.f_dentry <addr>
crash> struct dentry.d_inode <addr>
crash> struct inode.i_pipe <addr>

Batch Slab Inspection

crash> kmem -S inode_cache | grep counter | grep -v "= 1"

Kernel Linked List Traversal

crash> list task_struct.tasks -s task_struct.pid -h <start>
crash> list -h <addr> -s dentry.d_name.name

Extended Reference

For detailed information, refer to the following reference files:

Usage:

crash> help <command>        # Built-in help
# Or ask Claude to view reference files

Common Errors

crash: vmlinux and vmcore do not match!
# -> Ensure vmlinux version exactly matches vmcore

crash: cannot find booted kernel
# -> Specify vmlinux path explicitly

crash: cannot resolve symbol
# -> Check if vmlinux has debug symbols

Security Warnings

⚠️ Dangerous Operations

The following commands can cause system damage or data loss:

🔒 Sensitive Data Handling

• vmcore files contain complete kernel memory, potentially including:
  • User process memory and credentials
  • Encryption keys and secrets
  • Network connection data and passwords
• Access control: Restrict vmcore file access to authorized personnel
• Secure storage: Store dump files in encrypted or access-controlled directories
• Secure disposal: Use shred or secure delete when disposing of vmcore files

🛡️ Best Practices

1. Only analyze vmcore files in isolated/test environments when possible
2. Never share raw vmcore files publicly without sanitization
3. Consider using makedumpfile -d to filter sensitive pages before analysis
4. Document and audit all crash analysis sessions for compliance

Important Notes

1. Version Match: vmlinux must exactly match the vmcore kernel version
2. Debug Info: Must use vmlinux with debug symbols
3. Context Awareness: bt, files, vm commands are affected by current context
4. Live System Modification: wr command modifies running kernel, extremely dangerous

Resources

• Crash Utility Whitepaper
• Crash Utility Documentation
• Crash Help Pages

Contributing

This is an open-source project. Contributions are welcome!

• GitHub Repository: https://github.com/crazyss/linux-kernel-crash-debug
• Report Issues: GitHub Issues
• Submit PRs: Pull requests are welcome for bug fixes, new features, or documentation improvements

See CONTRIBUTING.md for guidelines.