Huawei Cloud CCE/UCS Workload Manager

Overview

This skill provides workload lifecycle management capabilities for Huawei Cloud CCE and UCS-managed Kubernetes clusters using hcloud CLI for kubeconfig acquisition and kubectl for Kubernetes resource operations.

Architecture: hcloud CLI → kubeconfig YAML → kubectl --kubeconfig → k8s resources (Deployment/StatefulSet/DaemonSet/Job/CronJob/Service/Ingress/ConfigMap/Secret/PVC)

Related Skills:

• cce-cluster-management - CCE cluster infrastructure creation, scaling, and deletion
• ucs-cluster-onboarding-manager - UCS cluster onboarding, lifecycle, and fleet grouping
• ucs-policy-governor - UCS policy governance, compliance, and audit management

Capabilities:

• Obtain kubeconfig for CCE clusters (direct cluster access)
• Obtain federation kubeconfig for UCS fleet (multi-cluster fleet operations)
• Manage Deployment/StatefulSet/DaemonSet lifecycle (create, query, scale, update, delete)
• Manage Job/CronJob lifecycle (create, query, suspend, resume, delete)
• Configure HPA autoscaling for Deployments and StatefulSets
• Manage Service/Ingress networking, ConfigMap/Secret configuration, PVC storage
• Observe Pod status, logs, and events
• Manage namespaces
• Install and configure kubectl

Typical Use Cases:

• "Get kubeconfig for my CCE cluster"
• "Deploy an application to CCE"
• "Scale my Deployment to 5 replicas"
• "Roll back a Deployment update"
• "Create a StatefulSet for my database"
• "Deploy a logging agent as DaemonSet"
• "Create a CronJob for scheduled tasks"
• "Set up HPA for auto-scaling"
• "Check Pod logs for debugging"
• "Create a Service and Ingress for my app"
• "Manage ConfigMaps and Secrets"
• "Create a PersistentVolumeClaim"
• "List namespaces in my cluster"
• "Install kubectl on my machine"

Prerequisites

1. hcloud CLI Requirements (MANDATORY)

• hcloud CLI installed (version >= 7.2.2)
• Run hcloud version to verify installation
• First-time usage: printf "y\n" | hcloud version to accept privacy statement

2. kubectl Requirements (MANDATORY)

• kubectl installed (version compatible with cluster Kubernetes version)
• See Task: kubectl Setup for installation guidance
• Run kubectl version --client to verify installation

3. Credential Configuration

• Valid Huawei Cloud credentials (AK/SK mode)
• Security Rules:
  • 🚫 Never expose AK/SK values in code, conversation, or commands
  • 🚫 Never use echo $HUAWEI_CLOUD_AK or echo $HUAWEI_CLOUD_SK to check credentials
  • ✅ Use environment variables: HUAWEI_CLOUD_AK, HUAWEI_CLOUD_SK, HUAWEI_CLOUD_REGION
  • ✅ Prefer IAM users over root account for cloud operations
  • ✅ Enable MFA for sensitive operations

Configuration Method (Environment Variables Only):

export HUAWEI_CLOUD_AK=<your-ak>
export HUAWEI_CLOUD_SK=<your-sk>
export HUAWEI_CLOUD_REGION=cn-north-4

⚠️ Important Security Notes:

• Never commit credentials to version control
• Use IAM users with minimal required permissions
• Enable MFA for sensitive operations
• Rotate AK/SK regularly

4. IAM Permission Requirements

Two-Layer Permission Model:

1. Huawei Cloud IAM: Controls kubeconfig acquisition via hcloud CLI (who can get a kubeconfig for CCE or UCS fleet)
2. Kubernetes RBAC: Controls kubectl operations after kubeconfig is obtained (what the user can do in the cluster or fleet)

Permission Failure Handling:

1. When hcloud commands fail due to IAM permission errors, verify the IAM permissions listed above
2. When kubectl commands fail due to RBAC permission errors, the cluster administrator must configure appropriate RBAC roles
3. Guide the user to create custom policies in the IAM console for Huawei Cloud permissions
4. Guide the user to create ClusterRole/Role bindings for Kubernetes RBAC permissions
5. Pause execution and wait for user confirmation that permissions have been granted

Core Commands

1. Kubeconfig Acquisition

See Task: Kubeconfig Acquisition for detailed workflows.

# CCE cluster kubeconfig (duration in days, 1-1827)
hcloud CCE CreateKubernetesClusterCert --cluster_id=<cce-cluster-id> --duration=1 --cli-region=cn-north-4

# UCS federation kubeconfig (duration in days, 1-1825)
# Used for multi-cluster fleet operations via UCS
hcloud UCS DownloadFederationKubeconfig --clustergroupid=<fleet-id> --duration=1 --cli-region=cn-north-4

Save kubeconfig and verify:

# Save CCE kubeconfig to file
hcloud CCE CreateKubernetesClusterCert --cluster_id=<cce-cluster-id> --duration=1 --cli-region=cn-north-4 > ~/.kube/cce-kubeconfig.yaml

# Verify connection
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml cluster-info

Key Parameter Differences:

• CCE uses --cluster_id (with underscore) for cluster ID
• UCS uses --clusterid (no underscore) for cluster ID
• CCE CreateKubernetesClusterCert --duration is in days (1-1827)
• UCS DownloadFederationKubeconfig --duration is in days (1-1825)

2. Workload Management (Deployment/StatefulSet/DaemonSet)

See Task: Deployment Management and Task: StatefulSet/DaemonSet Management for detailed workflows.

All commands use kubectl --kubeconfig=<kubeconfig-file> -n <namespace> pattern.

# Create Deployment from YAML file
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml apply -f deployment.yaml -n production

# Create Deployment inline
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml create deployment my-app --image=myapp:v1 --replicas=3 -n production

# Query Deployment status
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml get deployments -n production
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml describe deployment my-app -n production

# Scale Deployment
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml scale deployment my-app --replicas=5 -n production

# Update Deployment (rolling update)
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml set image deployment/my-app my-app=myapp:v2 -n production
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml rollout status deployment/my-app -n production

# Rollback Deployment
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml rollout undo deployment/my-app -n production

# Delete Deployment
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml delete deployment my-app -n production

# StatefulSet operations (same pattern)
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml apply -f statefulset.yaml -n production
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml scale statefulset my-db --replicas=3 -n production

# DaemonSet operations (same pattern)
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml apply -f daemonset.yaml -n production
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml rollout status daemonset/log-agent -n production

3. Job/CronJob Management

# Create Job from YAML
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml apply -f job.yaml -n production

# Create Job inline
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml create job data-migration --image=migrator:v1 -n production

# Query Job status
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml get jobs -n production
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml describe job data-migration -n production

# Delete Job
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml delete job data-migration -n production

# Create CronJob from YAML
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml apply -f cronjob.yaml -n production

# Create CronJob inline
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml create cronjob nightly-backup --image=backup:v1 --schedule="0 2 * * *" -n production

# Suspend CronJob
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml patch cronjob nightly-backup --type merge --patch-file=suspend.json -n production

# Resume CronJob
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml patch cronjob nightly-backup --type merge --patch-file=resume.json -n production

# Delete CronJob
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml delete cronjob nightly-backup -n production

4. Observability + Config

# Pod status
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml get pods -n production
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml get pods -o wide -n production

# Pod logs
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml logs my-app-pod -n production
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml logs my-app-pod --tail=100 -n production
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml logs my-app-pod -f -n production

# Pod events
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml describe pod my-app-pod -n production

# Namespace management
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml get namespaces
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml create namespace staging
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml delete namespace staging

# Service
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml apply -f service.yaml -n production
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml get services -n production

# Ingress
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml apply -f ingress.yaml -n production
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml get ingress -n production

# ConfigMap
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml create configmap app-config --from-literal=key1=value1 -n production
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml get configmaps -n production

# Secret
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml create secret generic db-secret --from-literal=password=s3cret -n production
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml get secrets -n production

# PVC
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml apply -f pvc.yaml -n production
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml get pvc -n production

# HPA autoscaling
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml autoscale deployment my-app --min=2 --max=10 --cpu=80% -n production
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml get hpa -n production

5. kubectl Installation & Configuration

See Task: kubectl Setup for detailed installation and configuration guidance.

Parameter Reference

hcloud Parameters (Kubeconfig Acquisition)

kubectl Flags

Output Format

CCE CreateKubernetesClusterCert (Kubeconfig YAML)

Returns a standard Kubernetes kubeconfig YAML document containing:

apiVersion: v1
kind: Config
clusters:
- cluster:
    certificate-authority-data: <base64-encoded-ca>
    server: https://<cluster-endpoint>:5443
  name: internalCluster
- cluster:
    server: https://<cluster-eip>:5443
    insecure-skip-tls-verify: true
  name: externalCluster
- cluster:
    certificate-authority-data: <base64-encoded-ca>
    server: https://<cluster-eip>:5443
  name: externalClusterTLSVerify
contexts:
- context:
    cluster: internalCluster
    user: user
  name: internal
- context:
    cluster: externalCluster
    user: user
  name: external
- context:
    cluster: externalClusterTLSVerify
    user: user
  name: externalTLSVerify
current-context: external

UCS DownloadFederationKubeconfig (Federation Kubeconfig YAML)

Returns a federation kubeconfig with two contexts for multi-cluster fleet operations:

apiVersion: v1
kind: Config
clusters:
- cluster:
    certificate-authority-data: <base64-encoded-ca>
    server: https://<fleet-name>.fleet.ucs.<region>.myhuaweicloud.com:5443
  name: federation
- cluster:
    certificate-authority-data: <base64-encoded-ca>
    server: https://<fleet-name>.fleet.ucs.<region>.myhuaweicloud.com:5443/apis/cluster.karmada.io/v1alpha1/clusters/*/proxy
  name: karmada-aggregated-apiserver
contexts:
- context:
    cluster: federation
    user: user
  name: federation
- context:
    cluster: karmada-aggregated-apiserver
    user: user
  name: karmada-aggregated-apiserver
current-context: federation

Federation vs Karmada context:

• federation: Operates on fleet-level resources (propagated workloads, fleet policies)
• karmada-aggregated-apiserver: Proxy access to individual member clusters via /clusters/<cluster-name>/proxy

Network prerequisite: UCS federation kubeconfig uses <fleet-name>.fleet.ucs.<region>.myhuaweicloud.com as the API server domain. This domain resolves via UCS VPC Endpoint (VPCEP). If DNS resolution fails, ensure your network can reach the UCS VPCEP (e.g., via VPC peering, VPN, or direct cloud network access).

### kubectl Output Formats

| `-o` flag        | Description                        | Use Case                                    |
| ---------------- | ---------------------------------- | ------------------------------------------- |
| (default)        | Tabular human-readable output      | Quick status check                          |
| `-o wide`        | Extended tabular with extra columns | Detailed Pod/IP info                       |
| `-o yaml`        | YAML format                        | Export/edit resource specs                  |
| `-o json`        | JSON format                        | Scripting/automation                        |
| `-o name`        | Resource name only                 | Quick list of names                         |

## Common Region IDs

| Region Name                    | Region ID        |
| ------------------------------ | ---------------- |
| North China - Beijing 4        | `cn-north-4`     |
| North China - Beijing 1        | `cn-north-1`     |
| East China - Shanghai 1        | `cn-east-3`      |
| East China - Shanghai 2        | `cn-east-2`      |
| South China - Guangzhou        | `cn-south-1`     |
| South China - Shenzhen         | `cn-south-4`     |
| Southwest China - Guiyang 1    | `cn-southwest-2` |
| Asia Pacific - Bangkok         | `ap-southeast-2` |
| Asia Pacific - Singapore       | `ap-southeast-1` |
| Asia Pacific - Hong Kong       | `ap-southeast-3` |
| Europe - Paris                 | `eu-west-0`      |

## Verification

### Kubeconfig Acquisition Verification

```bash
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml cluster-info
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml get nodes

Workload Deployment Verification

kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml get deployments -n production
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml rollout status deployment/my-app -n production
kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml get pods -o wide -n production

HPA Verification

kubectl --kubeconfig=~/.kube/cce-kubeconfig.yaml get hpa -n production

Best Practices

1. Namespace-First: Always specify -n <namespace> explicitly; never rely on default namespace
2. Kubeconfig Security: Store kubeconfig files with restricted file permissions (chmod 600); never expose them in public repositories or CI logs
3. Short-Duration Certificates: Use minimum --duration needed for CCE kubeconfig; prefer 1 day for interactive sessions
4. RBAC Alignment: Ensure Huawei Cloud IAM permissions and Kubernetes RBAC roles are aligned; IAM grants kubeconfig access, RBAC grants kubectl operations
5. YAML-Based Deployment: Prefer kubectl apply -f over inline kubectl create for reproducibility and auditability
6. Rollout Monitoring: Always check rollout status after updates; use rollout undo for emergency rollback
7. Namespace Isolation: Use separate namespaces for production, staging, and development workloads
8. HPA Baseline: Set HPA --min to your steady-state replica count; set --max based on resource budget

Reference Documents

Notes

• kubectl is the primary tool — all workload operations use kubectl --kubeconfig=<file> after kubeconfig acquisition via hcloud
• kubeconfig is a secret — treat it like a credential; never share or expose publicly
• RBAC governs kubectl access — even with a valid kubeconfig, Kubernetes RBAC controls what operations are permitted
• cce-cluster-management handles infrastructure — cluster creation, deletion, and node management belong to the cce-cluster-management skill
• Two-layer permission model — Huawei Cloud IAM controls kubeconfig acquisition, Kubernetes RBAC controls kubectl operations
• UCS is for fleet operations — UCS kubeconfig is only for federation (DownloadFederationKubeconfig), not single cluster (CreateClusterKubeconfig is out of scope for this skill)
• UCS federation kubeconfig provides two contexts — federation (fleet-level) and karmada-aggregated-apiserver (proxy to member clusters)
• UCS federation requires network access — fleet API server domain (<fleet>.fleet.ucs.<region>.myhuaweicloud.com) requires VPCEP access

Common Pitfalls