360 Secure SkillHub
360 安全技能中心

Weights & Biases Monitor

通过 Python 脚本调用 Weights & Biases API，检查训练运行状态与心跳停顿，检测梯度爆炸/消失、loss 曲线趋势，估算进度与 ETA，并支持运行间对比与实验健康报告输出（含 JSON 格式）。

chrisvoncsefalvay
chrisvoncsefalvay
热度 78.59
最近更新 2026-05-11 15:50:57
分类：数据依赖：代码执行 / 第三方 API应用场景：研究 / 报告撰写 / 数据处理 / 自动化脚本

安装技能

复制下面的话术发给你的 Agent，它会通过 secure-skills 检索并安装该技能。

执行命令安装此 skill： npx secure-skills-cli add wandb-monitor
Metadata
name
wandb
description
Monitor and analyze Weights & Biases training runs. Use when checking training status, detecting failures, analyzing loss curves, comparing runs, or monitoring experiments. Triggers on "wandb", "training runs", "how's training", "did my run finish", "any failures", "check experiments", "loss curve", "gradient norm", "compare runs".

Weights & Biases

Monitor, analyze, and compare W&B training runs.

Setup

wandb login
# Or set WANDB_API_KEY in environment

Scripts

Characterize a Run (Full Health Analysis)

~/clawd/venv/bin/python3 ~/clawd/skills/wandb/scripts/characterize_run.py ENTITY/PROJECT/RUN_ID

Analyzes:

  • Loss curve trend (start → current, % change, direction)
  • Gradient norm health (exploding/vanishing detection)
  • Eval metrics (if present)
  • Stall detection (heartbeat age)
  • Progress & ETA estimate
  • Config highlights
  • Overall health verdict

Options: --json for machine-readable output.

Watch All Running Jobs

~/clawd/venv/bin/python3 ~/clawd/skills/wandb/scripts/watch_runs.py ENTITY [--projects p1,p2]

Quick health summary of all running jobs plus recent failures/completions. Ideal for morning briefings.

Options:

  • --projects p1,p2 — Specific projects to check
  • --all-projects — Check all projects
  • --hours N — Hours to look back for finished runs (default: 24)
  • --json — Machine-readable output

Compare Two Runs

~/clawd/venv/bin/python3 ~/clawd/skills/wandb/scripts/compare_runs.py ENTITY/PROJECT/RUN_A ENTITY/PROJECT/RUN_B

Side-by-side comparison:

  • Config differences (highlights important params)
  • Loss curves at same steps
  • Gradient norm comparison
  • Eval metrics
  • Performance (tokens/sec, steps/hour)
  • Winner verdict

Python API Quick Reference

import wandb
api = wandb.Api()

# Get runs
runs = api.runs("entity/project", {"state": "running"})

# Run properties
run.state      # running | finished | failed | crashed | canceled
run.name       # display name
run.id         # unique identifier
run.summary    # final/current metrics
run.config     # hyperparameters
run.heartbeat_at # stall detection

# Get history
history = list(run.scan_history(keys=["train/loss", "train/grad_norm"]))

Metric Key Variations

Scripts handle these automatically:

  • Loss: train/loss, loss, train_loss, training_loss
  • Gradients: train/grad_norm, grad_norm, gradient_norm
  • Steps: train/global_step, global_step, step, _step
  • Eval: eval/loss, eval_loss, eval/accuracy, eval_acc

Health Thresholds

  • Gradients > 10: Exploding (critical)
  • Gradients > 5: Spiky (warning)
  • Gradients < 0.0001: Vanishing (warning)
  • Heartbeat > 30min: Stalled (critical)
  • Heartbeat > 10min: Slow (warning)

Integration Notes

For morning briefings, use watch_runs.py --json and parse the output.

For detailed analysis of a specific run, use characterize_run.py.

For A/B testing or hyperparameter comparisons, use compare_runs.py.