CCE Auto Remediation Runner

⚠️ Execution Method (Must Read): This skill executes remediation actions via local Python scripts using the scripts/huawei-cloud.py dispatcher. Using hcloud, kubectl, or other CLI tools or direct API calls is prohibited.

• All actions are dispatched through scripts/huawei-cloud.py with --action <action_name> and --params <json_params>
• All scripts and environment check scripts are inside the skill package. You must use skill action=exec to execute them; do not run them directly in a shell
• For action names and parameters, see the Core Tools section below
• Do not attempt hcloud, kubectl, curl IAM, or other CLI/API methods. This skill does not depend on these tools
• All paths are relative to the skill directory, which is the directory where this SKILL.md resides

Overview

This skill converts remediation intent into reviewable, confirmable, verifiable execution plans. It operates in preview-first mode by default — all mutation actions require preview without confirm=true, explicit user confirmation of action/object/risks, then execution with confirm=true, followed by read-only verification.

This skill is applicable to the following scenarios:

1. Remediation actions triggered by root-cause analysis conclusions (e.g., Deployment rollback for CrashLoop/ImagePull/CommandNotFound)
2. Node operations: cordon, uncordon, drain, reboot ECS
3. Workload operations: scale, resize, rollback, delete
4. Node pool operations: resize node pool
5. Cluster operations: hibernate, awake
6. Security operations: HSS vulnerability status change
7. Auto-remediation orchestration via huawei_auto_remediation_run for multi-step remediation plans
8. Traffic cutover: bind/unbind cluster EIP
9. ECS instance operations: start, stop

This skill does NOT handle the following:

1. Read-only diagnosis (use huawei-cloud-cce-root-cause-analyzer or domain-specific diagnoser skills)
2. Auto-executing remediation without preview and user confirmation
3. Guessing or fabricating remediation results without evidence
4. Batch or fuzzy-target remediation without explicit user confirmation per object

Prerequisites

Before using, you must run the environment check script to complete environment validation and dependency installation in one step:

• Linux / macOS: skill action=exec: bash skill://scripts/check_env.sh
• Windows: skill action=exec: powershell -ExecutionPolicy Bypass -File skill://scripts/check_env.ps1

Windows Note: Do not use && to chain commands (PowerShell 5.x does not support it). Use semicolons if you need to change directories first.

The script will check in sequence: Python >= 3.6 → install dependencies → validate SDK → validate credentials → validate service availability. If the environment check fails, fix the issues before continuing with other actions.

Environment Variables:

Security Constraints:

1. Never persist credentials (AK/SK/Token/Certificate) to the filesystem
2. AK/SK exist only within the current request call stack; released after use
3. Only non-sensitive project IDs are cached in process memory (never written to disk)
4. All temporary certificate files must be deleted immediately after use
5. Never expose AK/SK in logs, responses, or error messages

Do not output the values of the above environment variables.

IAM Permission Requirements

Permission Failure Handling:

1. When any command fails due to permission errors, display required permission list and policy JSON
2. Guide the user to create a custom policy in the IAM console and grant authorization
3. Pause execution and wait for user confirmation that permissions have been granted

Core Tools

All actions are dispatched through scripts/huawei-cloud.py using skill action=exec.

Auto-Remediation Orchestration

Workload Actions

Node Actions

Node Pool and Cluster Actions

ECS Instance Actions

Elastic Scaling Policy

Network / Traffic Actions

Security Actions

Verification (Read-Only) Actions

Parameter Reference

Common Parameters:

*Required for specific actions as noted.

Optional Parameters (passed via --params JSON):

Output Format

Remediation Preview (confirm=false)

{
  "success": false,
  "requires_confirmation": true,
  "remediation_trace_id": "ARR-...",
  "strategy": "rollback_previous_revision",
  "diagnosis": {},
  "action_result": {},
  "preview": {
    "action": "huawei_rollback_cce_workload",
    "target": {
      "region": "cn-north-4",
      "cluster_id": "cluster-id",
      "namespace": "default",
      "kind": "Deployment",
      "name": "app-server"
    },
    "current_state": {},
    "expected_state": {},
    "impact_scope": {},
    "rollback_method": "Re-apply current revision"
  },
  "risk_level": "R2",
  "rollback_notes": [],
  "summary": "Remediation plan preview — requires user confirmation before execution"
}

Remediation Execution (confirm=true)

{
  "success": true,
  "requires_confirmation": false,
  "confirmation_received": true,
  "remediation_trace_id": "ARR-...",
  "strategy": "rollback_previous_revision",
  "action_result": {},
  "execution": {
    "action": "huawei_rollback_cce_workload",
    "timestamp": "...",
    "result": {}
  },
  "verification": [
    {
      "method": "huawei_get_cce_pods",
      "status": "healthy",
      "details": {}
    }
  ],
  "report_markdown": "# CCE Auto Remediation Execution Report...",
  "report_file": "optional"
}

Full Auto-Remediation Orchestration Output

{
  "success": false,
  "requires_confirmation": true,
  "remediation_trace_id": "ARR-...",
  "strategy": "rollback_previous_revision",
  "diagnosis": {},
  "action_result": {},
  "verification": {},
  "summary": "remediation plan or execution result",
  "action": "huawei_auto_remediation_run",
  "risk_level": "R2",
  "target": {
    "region": "cn-north-4",
    "cluster_id": "optional",
    "resource": "optional"
  },
  "preview": {},
  "requires_confirmation": true,
  "confirmation_received": false,
  "execution": {},
  "verification": [],
  "rollback_notes": [],
  "report_markdown": "# CCE Auto Remediation Execution Report...",
  "report_file": "optional"
}

Verification

1. Run the environment check script to confirm dependencies and credentials are available
2. Use huawei_cce_node_cordon (without confirm=true) on a test node to verify preview mode returns requires_confirmation: true
3. After user confirmation, execute with confirm=true and verify node status with huawei_get_kubernetes_nodes
4. Use huawei_rollback_cce_workload preview mode to verify it shows current vs expected state
5. After rollback execution, verify workload health with huawei_workload_rollout_diagnose
6. Use huawei_auto_remediation_run preview mode to verify multi-step orchestration plan is shown before execution
7. Confirm that all R3 actions (drain, reboot, delete, hibernate) require explicit user confirmation
8. Verify that post-execution verification actions return healthy/expected status

Best Practices

1. Always preview first: Never call any mutation action with confirm=true on the first invocation. Always preview without confirm=true first
2. State the four essentials: Before confirmation, restate the action, object, parameters, impact scope, and rollback plan to the user
3. Prefer rollback for deployment failures: If root cause is from huawei-cloud-cce-root-cause-analyzer and involves startup command, CrashLoop, probe, or image causing new version unavailability, prefer huawei_auto_remediation_run with rollback_previous_revision strategy
4. Verify after execution: Every execution must be followed by read-only verification (Pod status, Node status, Events, workload rollout diagnosis)
5. Classify risk correctly: Refer to references/risk-rules.md for R1/R2/R3 classification; apply appropriate confirmation requirements
6. Never auto-add confirm: Deployment rollback, scale, resize, resource modification, delete cluster/node/workload, drain, reboot, and HSS vulnerability status change must all be preview → user confirm → execute → verify
7. Use auto-remediation orchestration for multi-step plans: When remediation involves multiple actions, use huawei_auto_remediation_run to produce a complete execution report with diagnosis basis, action results, and verification results
8. Cross-skill handoff for diagnosis: When root cause analysis is needed before remediation, hand off to huawei-cloud-cce-root-cause-analyzer; this skill only executes confirmed remediation actions
9. Document rollback notes: Every execution plan must include rollback method — how to revert if the remediation causes unintended effects

Reference Documents

• Workflow and action orchestration steps: references/workflow.md
• Risk classification and confirm=true rules: references/risk-rules.md
• Output execution record schema: references/output-schema.md
• Huawei Cloud CCE Documentation
• Huawei Cloud Python SDK Documentation

Notes

1. This skill is a MUTATION skill — it performs write actions (drain, cordon, scale, restart, delete, reboot, hibernate, vulnerability status change). Preview+confirm workflow is mandatory
2. Do not output the values of HW_ACCESS_KEY, HW_SECRET_KEY, HW_SECURITY_TOKEN, or other environment variables
3. All scripts must be executed via skill action=exec; do not run them directly in a shell
4. NEVER auto-add confirm=true. User must explicitly confirm the specific action, object, and risks
5. The environment check script must be run before any remediation action
6. When using temporary AK/SK, HW_SECURITY_TOKEN must be set
7. After execution, must call read-only verification actions to confirm status
8. Cross-skill references: diagnosis → huawei-cloud-cce-root-cause-analyzer; domain-specific diagnosis → huawei-cloud-cce-pod-failure-diagnoser, huawei-cloud-cce-node-failure-diagnoser, huawei-cloud-cce-network-failure-diagnoser

Common Pitfalls

1. Auto-adding confirm=true — The most critical pitfall. NEVER assume user intent implies confirmation. Always preview first, show results, and wait for explicit user confirmation
2. Skipping preview for R2 actions — Even medium-risk actions (scale, resize, cordon, rollback) require preview. No mutation action may skip the preview step
3. Not verifying after execution — Every R2/R3 execution must be followed by read-only verification (Pod/Node/Workload/Events status). Skipping verification leaves remediation unconfirmed
4. Batch or fuzzy-target remediation — R3 actions (drain, reboot, delete, hibernate) must have explicit, specific target objects. Never execute with vague or batch targets without per-object confirmation
5. Not documenting rollback method — Every remediation plan must state how to revert if the action causes unintended effects. Omitting rollback notes is a safety hazard
6. Executing remediation without diagnosis — Always confirm root cause via huawei-cloud-cce-root-cause-analyzer or domain diagnoser before remediation. Blind remediation without evidence is prohibited
7. Confusing R2 and R3 risk levels — R2 (runtime impact) requires preview+confirm; R3 (destructive) requires explicit per-object confirmation with additional verification. See references/risk-rules.md
8. Not restating the plan to the user — Before requesting confirmation, restate the action, target object, region, cluster_id, expected impact, and rollback plan. The user must confirm all four essentials