Huawei Cloud CCE AZ Power Outage Chaos Experiment

Overview

End-to-end CCE AZ power outage chaos experiment in three phases:

Phase 1: Prepare          Phase 2: Execute           Phase 3: Log Analysis
─────────────             ──────────────             ──────────────────
Discover clusters    →    Pre-check (nodes Ready)  → Load experiment context
Select cluster            Shutdown (BatchStop)       Discover dependencies
Discover node AZs         Monitor (Node/Pod)         Collect logs (LTS + Pod)
Select target AZ          Wait (duration)            Analyze error patterns
Validate compatibility    Rollback (BatchStart)      Generate analysis report
Generate + deploy config  Verify + report

Scenario: Shut down all CCE nodes in one AZ → Kubernetes marks nodes NotReady → Pods evicted and rescheduled to other AZs → start nodes back → verify recovery. Tests cluster HA under AZ-level failure.

Prerequisites

• hcloud CLI — installed and configured with AK/SK (ECS + CCE permissions)
• kubectl — installed and configured for CCE cluster access
• Python 3 — for discovery, validation, execution, and analysis scripts
• Environment variables: HW_ACCESS_KEY, HW_SECRET_KEY, HW_REGION_NAME (optional, defaults to cn-north-4 if unset)

Run environment check:

bash scripts/check_env.sh

If kubectl is missing, check_env.sh auto-installs it. Manual install:

ARCH=$(uname -m | sed 's/x86_64/amd64/;s/aarch64/arm64/')
curl -fsSL "https://dl.k8s.io/release/v1.28.0/bin/linux/${ARCH}/kubectl" -o /tmp/kubectl_download
sleep 1 && mv /tmp/kubectl_download /usr/local/bin/kubectl && chmod 755 /usr/local/bin/kubectl

Phase 1: Prepare (故障演练准备)

Generates experiment configuration. Does NOT start the experiment.

Step 1.1: Environment Check

bash scripts/check_env.sh

Step 1.2: Determine Region [INTERACTIVE]

Check HW_REGION_NAME environment variable:

• Set → use that region (confirm with user: "Region is , correct?").
• Not set → ask the user which region to use. Do NOT silently default to cn-north-4.

echo $HW_REGION_NAME

Region is used in all subsequent --region parameters and hcloud --cli-region calls.

Step 1.3: Discover CCE Clusters

python3 scripts/discover_cce.py --region <region>

Output: JSON with all clusters (ID, name, status, flavor, version).

Step 1.4: User Selects Cluster [INTERACTIVE]

Present discovered clusters in a table. Ask the user to choose — never auto-select.

• Only Available clusters can be selected directly.
• Hibernation clusters must be awakened first.

Step 1.5: Get Credentials & Discover Node AZ Distribution

# Obtain kubeconfig
hcloud CCE CreateKubernetesClusterCert --cluster_id=<id> --cli-region=<region> --duration=30 --cli-output=json > /root/.kube/config

# Discover node AZ distribution
KUBECONFIG=/root/.kube/config kubectl get nodes --show-labels

Present node-to-AZ mapping to the user.

Step 1.6: User Selects Target AZ [INTERACTIVE]

Present AZ distribution. Ask the user to choose the target AZ.

• Show node count per AZ.
• Warn if cluster has only one AZ (no cross-AZ redundancy).
• Never auto-select.

Step 1.7: Create Experiment Directory

Create the experiment directory before running discovery and validation, so all artifacts (discovery.json, validation.json, experiment.json, logs, reports) live inside one directory.

EXP_DIR="./experiments/$(date -u +%Y%m%d-%H%M%S)-cce-az-power-$(echo <az> | tr -d '-')"
mkdir -p "$EXP_DIR"
echo "$EXP_DIR"

All subsequent steps use $EXP_DIR as the output path.

Step 1.8: Discover Nodes & Pods in Target AZ

KUBECONFIG=/root/.kube/config python3 scripts/discover_cce.py \
    --region <region> --cluster-id <id> --az <az> \
    --include-pods --output "$EXP_DIR/discovery.json"

Step 1.9: Validate Compatibility [CRITICAL GATE]

KUBECONFIG=/root/.kube/config python3 scripts/validate_targets.py \
    --discovery-file "$EXP_DIR/discovery.json" \
    --cluster-id <id> --az <az> --region <region> \
    --output "$EXP_DIR/validation.json"

Validation rules (see references/cce-validation-rules.md):

1. All target nodes must be Ready
2. Cross-AZ capacity sufficient for rescheduling
3. PDB constraints checked (warnings only for AZ outage — shutdown bypasses eviction API)
4. Workload replicas ≥ 2 (warning)
5. Single-AZ cluster risk (error)

Decision: errors → stop and fix. warnings only → proceed to generate.

Step 1.10: Generate, Deploy & Report

# Generate experiment configuration (use --experiment-dir to reuse $EXP_DIR)
python3 scripts/generate_experiment.py \
    --discovery-file "$EXP_DIR/discovery.json" --validation-file "$EXP_DIR/validation.json" \
    --cluster-id <id> --az <az> --region <region> \
    --duration 300 --experiment-dir "$EXP_DIR"

# Deploy experiment (local mode, generates emergency rollback script)
bash scripts/deploy_experiment.sh --experiment-dir "$EXP_DIR/"

Experiment is prepared but NOT started. Proceed to Phase 2 when ready.

Phase 1 References

• references/cce-az-power-scenario.md — scenario definition, fault impact chain, API mapping
• references/cce-validation-rules.md — validation rules with rationale and decision matrix
• references/experiment-template-guide.md — experiment.json schema with full example

Phase 2: Execute (故障演练执行)

Performs actual node shutdown. Target CCE nodes will be stopped.

Step 2.1: Environment Check

bash scripts/check_env.sh

Step 2.2: Execute Experiment [Core Step]

# Normal execution
python3 scripts/execute_experiment.py --experiment-dir "$EXP_DIR/"

# Dry Run (simulate without API calls)
python3 scripts/execute_experiment.py --experiment-dir "$EXP_DIR/" --dry-run

# Auto-rollback on shutdown failure
python3 scripts/execute_experiment.py --experiment-dir "$EXP_DIR/" --auto-rollback

6 Execution Phases

Produces execution-log.json with complete timeline.

Step 2.3: Generate Report

python3 scripts/generate_report.py --log-file "$EXP_DIR/execution-log.json"

Emergency Rollback

python3 scripts/rollback_experiment.py --experiment-dir "$EXP_DIR/"
# Or specify instance IDs directly
python3 scripts/rollback_experiment.py --ids "id1,id2" --region <region>

Phase 2 References

• references/execution-workflow.md — 6-phase details, execution-log.json format, safety mechanisms
• references/monitoring-guide.md — Node/Pod state machines, polling mechanism

CCE Resource State Machine

Node:  Ready ──关机──→ NotReady ──开机──→ Ready
Pod:   Running → Terminating → Pending → Running (on new node in other AZ)

Phase 3: Log Analysis (生成日志分析报告)

Collects and analyzes application/CCE logs within the experiment time window.

Step 3.1: Environment Check

bash scripts/check_env.sh

Step 3.2: Load Experiment Context & Analyze Logs

python3 scripts/analyze_logs.py --experiment-dir "$EXP_DIR/" \
    --output-dir "$EXP_DIR/log-analysis"

Auto-detects mode:

• execution-log.json exists → post-hoc analysis (事后分析)
• Only experiment.json exists → real-time monitoring (实时监控)

Step 3.3: Discover Affected Services

# Extract affected node names from discovery.json or use known nodes
python3 scripts/discover_dependencies.py \
    --nodes "192.168.0.49,..." \
    --output "$EXP_DIR/log-analysis/dependencies.json"

Scans Service/Ingress/ConfigMap references to identify directly impacted services. Note: discover_dependencies.py queries the cluster directly via kubectl, not from discovery.json. Pass the affected node names with --nodes.

Tip: Step 3.2 (analyze_logs.py) already calls discover_dependencies.py internally, so this step is optional — use it only for standalone dependency discovery.

Step 3.4: Collect Logs

python3 scripts/collect_logs.py --experiment-dir "$EXP_DIR/" \
    --output-dir "$EXP_DIR/log-analysis/collected_logs"

Collects within experiment time window:

• CCE Pod logs: kubectl logs --since-time
• LTS logs: hcloud LTS ListLogs
• Kubernetes Events: kubectl get events

Step 3.5: Generate Analysis Report

python3 scripts/generate_analysis_report.py \
    --analysis "$EXP_DIR/log-analysis/analysis-result.json" \
    --context "$EXP_DIR/log-analysis/dependencies.json" \
    --output "$EXP_DIR/log-analysis/analysis-report.md"

Output: Markdown report with:

• Experiment overview & affected services
• Pod rescheduling timeline
• Error pattern statistics (ERROR/Exception/Connection refused/Timeout/5xx)
• Business impact assessment
• Improvement recommendations

Phase 3 References

• references/analysis-workflow.md — 6-step analysis workflow, error pattern classification table
• references/managed-service-logs.md — kubectl logs, LTS, CES commands, best practices

Error Pattern Classification

Script Reference

Key Design Decisions

1. Interactive cluster and AZ selection — the agent must present discovered clusters and AZ distributions to the user and let the user choose. Never auto-select.

2. Validate before generating — cross-AZ capacity and PDB constraints are checked before any files are produced.

3. PDB warnings are informational for AZ outage — PDBs govern voluntary pod eviction, not direct node shutdown via ECS API. PDB violations are warnings, not errors.

4. Dry Run and auto-rollback — --dry-run simulates without API calls; --auto-rollback triggers rollback on shutdown failure.

5. Local deployment mode — deploy_experiment.sh generates the emergency rollback script locally; the full drill flow is executed by the built-in scripts/execute_experiment.py (no external COC dependency).

6. AZ-scoped targeting — targets all CCE nodes in a specified AZ, simulating real AZ power outage.

7. Default duration 5 minutes (300s) — sufficient for observing pod rescheduling while limiting blast radius.

8. Two analysis modes — post-hoc (from execution-log.json) and real-time (from experiment.json during experiment).

9. All artifacts in one experiment directory — $EXP_DIR is created before discovery; discovery.json, validation.json, experiment.json, execution-log.json, and log-analysis/ all live inside it.

Known Script Issues & Fixes

1. discover_cce.py — invalid kubeconfig detection: hcloud CCE DownloadClusterConfig returns error with exit code 0 on unsupported versions. Fix: validate config contains "apiVersion" or "clusters"; use CreateKubernetesClusterCert as fallback.

2. discover_cce.py — get_nodes_by_az return value: Returned [] instead of [], [] on kubectl failure, causing ValueError. Fix: return [], [].

3. validate_targets.py — CPU millicore parsing: float("1930m") caused ValueError. Fix: parse_cpu() / parse_memory() helpers handle m, Ki, Mi, Gi suffixes.

4. discover_cce.py — wrong ECS instance ID: Extracted from spec.providerID which contains CCE-internal node ID, not ECS server ID. Fix: get_ecs_instance_map(region) queries hcloud ECS ListServersDetails and builds private-IP → ECS-ID map.

5. deploy_experiment.sh + execute_experiment.py + rollback_experiment.py — unsupported --body parameter: hcloud KooCLI does not support --body for ECS APIs. Fix: use native parameter format --os-stop.servers.1.id=xxx --os-stop.type=SOFT and --os-start.servers.N.id=xxx.

Limitations

• Depends on hcloud CLI with ECS and CCE permissions
• Depends on kubectl with CCE cluster access
• Cross-AZ capacity check based on node allocatable resources, not real-time usage
• Pod logs only from currently alive Pods; deleted Pod logs unavailable
• LTS log queries require LTS agent (ICAgent) installed in CCE cluster
• Focused on single AZ power outage — does not handle composite scenarios