VMware AIops

Disclaimer: This is a community-maintained open-source project and is not affiliated with, endorsed by, or sponsored by VMware, Inc. or Broadcom Inc. "VMware" and "vSphere" are trademarks of Broadcom. Source code is publicly auditable at github.com/vmware-skills/VMware-AIops under the MIT license.

VMware family entry point — AI-powered VM lifecycle, deployment, and alarm management — 60 MCP tools.

Start here: install vmware-aiops first, then add modules as needed. Run vmware-aiops hub status to see which family members are installed. Family: vmware-monitor (inventory/health), vmware-storage (iSCSI/vSAN), vmware-vks (Tanzu Kubernetes), vmware-nsx (NSX networking), vmware-nsx-security (DFW/firewall), vmware-aria (metrics/alerts/capacity), vmware-avi (AVI/ALB/AKO), vmware-harden (compliance baselines). | vmware-pilot (workflow orchestration) | vmware-policy (audit/policy)

What This Skill Does

Audit & Safety

Read before connecting an agent. Per-tool inventory: references/capabilities.md.

• MCP gates default to a no-write preview. 23 write tools — every destructive one — return their blast radius unless confirm=True, which is refused on a blocker or unreadable measurement; delete_vm also needs acknowledge_blast_radius echoing the preview. The other 20 (create, clone, deploy, power-on, reconfigure) act on the first call.
• The enforcement boundary is vCenter/ESXi RBAC: an agent can do whatever the configured account can. Use a dedicated, least-privilege service account scoped to what the agent may change (a read-only role makes the skill read-only). Store its password in ~/.vmware-aiops/.env (0600) or a secret manager (VMWARE_<TARGET>_PASSWORD).
• CLI only: destructive commands require double confirmation; most CLI writes take --dry-run. Neither applies to MCP.
• Policy: deny rules and a maintenance window in ~/.vmware/rules.yaml are checked before every MCP and remote CLI call (e.g. deny writes to environment: production targets). The shipped baseline denies nothing. An in-process guardrail, not a substitute for RBAC.
• Audit: every MCP call is recorded in ~/.vmware/audit.db, credentials redacted (vmware-audit log --last 20). Best-effort: a failed audit write warns, never blocks.
• Guest ops run any command or file write the guest account allows — a read-only vCenter role does not limit this. username is required (no default account) and over MCP the password is a tool argument the agent sees; pass a minimal guest account. upload_vm_guest_file reads any local file the server can read.

Quick Install

uv tool install vmware-aiops==2.1.0
vmware-aiops doctor
vmware-aiops hub status   # see which family members are installed

VMware Family — Install What You Need

vmware-aiops is the entry point. Add modules for additional capabilities:

Each module stays independent — small tool count keeps local models (Ollama, Qwen) accurate.

When to Use This Skill

• Power on/off, create, delete, snapshot, clone, or migrate VMs
• Deploy VMs from OVA, templates, linked clones, or batch specs
• Run commands or transfer files inside a VM (Guest Operations)
• Create/configure clusters (HA/DRS)
• Browse datastores for deployable images
• Plan and execute multi-step operations with rollback
• List, acknowledge, and clear vCenter triggered alarms (clear matches by entity type + status — see MCP Tools section)

Use companion skills for:

• Inventory, health, alarms, VM info → vmware-monitor
• iSCSI, vSAN, datastore management → vmware-storage
• Tanzu Kubernetes (Supervisor, Namespace, TKC) → vmware-vks
• Load balancing, AVI/ALB, AKO, Ingress → vmware-avi

Related Skills — Skill Routing

Common Workflows

Diagnostic investigations: Before remediating any "why is X slow / failing / down" issue, follow references/investigation-protocol.md. It enforces the four root-cause completeness criteria (falsifiability / sufficiency / necessity / mechanism) and the up-to-three-rounds deepening loop. Only invoke L3+ write tools after the four criteria are satisfied AND the user has approved a remediation plan.

Cluster Health Triage ("what's wrong right now?")

Start here when the ask is "is anything on fire?" before diving into a specific VM. This is a read-only rollup delegated to vmware-monitor, exposed here so triage-then-act stays in one conversation.

1. One glance --> get_cluster_health_summary (MCP) or vmware-aiops summary (CLI). Read top_issues first — ranked anomalies (disconnected hosts, red/yellow alarms, capacity pressure), each with a drill-down hint; the per-cluster table is context
2. Act on what it surfaces --> a host_down row → investigate the host; an alarm → acknowledge_vcenter_alarm / reset_vcenter_alarm; a hot VM implicated → migrate_vm or reconfigure_vm (after the investigation protocol)
3. Save/share a snapshot --> vmware-aiops summary --html writes an offline, timestamped HTML file (identical to vmware-monitor summary --html — same shared renderer)
4. If vmware-monitor is not installed --> this command/tool is unavailable (AIops delegates to it); install vmware-monitor, or use the deeper per-object read tools in that skill

Object-Centered Investigation → Act (drill-down before you change anything)

Judgment: after triage points at a problem object, drill in with one correlated read before actuating — the bundle aggregates the object with its surrounding infrastructure and recent history so you (and the operator) see the full picture, not a guess. AIops is the conversational entry point, so triage → investigate → act stays in one conversation.

1. Estate-wide --> get_cross_vcenter_attention ("what needs attention now?" across every vCenter). One vCenter configured → skip to get_cluster_health_summary
2. Drill into the flagged object (offer the level; skip the question when unambiguous):
   • a VM --> investigate_vm → state, host it runs on, cluster context, backing datastores, snapshots, alarms & recent changes, performance signals, correlated event timeline
   • a host --> investigate_host; a datastore --> investigate_datastore
3. Then act on the evidence --> e.g. bundle shows a wedged VM on a hot host → migrate_vm; a full datastore → delete_vm_snapshot on the sprawl the bundle surfaced. Follow the investigation protocol before any destructive action
4. Widen the window with hours=72; render an offline snapshot with --html (drill-down sections collapse natively, nothing uploaded)
5. If the object name is unknown --> the bundle returns a teaching error naming how to list objects; get the exact name and retry. If vmware-monitor is not installed --> these delegated tools are unavailable

Deploy a Lab Environment

Pre-flight (judgment, not blind sequence):

• Free space: target datastore must have ≥ OVA size × 2 (delta files + thin-provision overhead). If multiple datastores qualify, prefer one with lowest current IOPS pressure (cross-check vmware-aria if available).
• Name hygiene: prefix with date or owner (lab-2026-04-30-alice) so the TTL cleanup audit trail is meaningful.
• TTL: always set. 480 min for a single test session, 7200 min for a week-long sandbox. Never deploy a "lab" VM without a TTL — that is how datastores fill up at 3 AM.
• Snapshot timing: take the baseline after provisioning succeeds, not before — a pre-provision snapshot is just an empty checkpoint.

Steps:

1. vmware-aiops datastore browse <ds> --pattern "*.ova" → confirm image present and size
2. vmware-aiops deploy ova <path> --name <date>-<owner>-<purpose> --datastore <ds>
3. vmware-aiops vm guest-exec <name> --cmd /usr/bin/python3 --args "setup.py" --user admin → if exit ≠ 0, stop, do not snapshot a half-provisioned VM
4. vmware-aiops vm snapshot-create <name> --name baseline (only if multi-iteration testing; skip for one-shot)
5. vmware-aiops vm set-ttl <name> --minutes 480

Batch Clone for Testing

Pre-flight:

• Source VM state: powered-off is safest. If powered-on, VMware Tools must be running and quiesce-capable, else clones may have inconsistent disk state.
• Capacity math: free_space ≥ source.size × count × 1.2 (full clone) or ≥ count × 2 GB (linked clone, delta-only).
• Decision rule: count > 10 → use linked clones (deploy linked-clone); seconds vs minutes per clone, ~100× less storage. Tradeoff: linked clones depend on source snapshot — deleting the snapshot breaks all children.
• Network exhaustion: each clone gets a unique MAC from the vSphere pool; if you batch > 200, verify pool capacity in advance.
• TTL: every clone must have one. Use the plan's metadata to track ownership.

Steps:

1. create_vm_plan with clone + reconfigure + set-ttl steps grouped per VM (atomic per clone)
2. Review the plan with the user — surface count, datastore, irreversible warnings
3. apply_vm_plan — stops on first failure (intentional, do not auto-resume)
4. On failure: rollback_vm_plan → reverses completed clones; manually verify rollback before retrying

Migrate VM to Another Host

Pre-flight (ALL must pass before issuing migrate):

• CPU compatibility: target host CPU family must match source, OR cluster must be in EVC mode. Live migration across mismatched CPUs fails mid-flight and may leave the VM stunned.
• Network parity: every portgroup the VM uses must exist on the target host's vSwitch with the same VLAN. Missing portgroup → vNICs disconnected post-migration.
• Storage visibility: target host must see all of the VM's datastores; otherwise this is a Storage vMotion, not a host migration — different (slower) operation.
• Affinity rules: if the VM is pinned to source by a DRS host-affinity rule, migration silently violates intent. Check cluster info first.
• Hardware passthrough: VMs with PCI passthrough (GPU, USB) cannot live-migrate — schedule a cold migration window.

Steps:

1. Verify VM state and current host via vmware-monitor vm info <name>
2. Verify target host: same cluster, EVC compatible, has required networks/datastores
3. vmware-aiops vm migrate <name> --to-host <target> — wait for task completion, do not assume success on return
4. Post-check: vm info confirms new host AND power state unchanged AND vNICs connected

Usage Mode

MCP Tools (60 — 17 read, 43 write)

List envelope: the read list tools — browse_datastore, list_vcenter_alarms, list_vm_plans, list_vm_snapshots, list_vm_ttls — return {items, returned, limit, total, truncated, hint} rather than a bare array. Read the rows from items and check truncated before concluding a listing is complete; empty items with truncated: false means checked-and-none, not a failure. The write batch_* tools keep their bare list (complete by construction). Rationale, total semantics, error shape: references/capabilities.md.

Read/write split: 17 tools are read-only (per [READ] docstring marker), 43 modify state — gating in Audit & Safety. set_vm_ttl schedules an unattended auto-delete.

Network write gating: all four network writes are preview/confirm-gated — confirm=False (default) returns the exact spec without writing. remove_host_vmk is fail-closed: it refuses when the vmk is selected for a host service (management/vMotion/vSAN), lives on a non-default netstack (NSX TEPs, dedicated vMotion stacks), carries a default gateway route, or when any of that cannot be verified — pass force_unprotected=True to override the non-absolute protections. The host's only management-enabled vmk is never removable (no override). set_vmk_service is fail-closed too: it refuses both directions when the host's service map is unreadable, and refuses (no override) to untag management from the host's only management-enabled vmk — the call rides the interface it would untag.

DRS rule gating: set_drs_rule_enabled, create_drs_rule, delete_drs_rule are preview/confirm-gated and idempotent (matching state returns a no-write noop). create_drs_rule handles VM-VM affinity/anti-affinity only (≥2 distinct VMs, all cluster members); VM-Host rules are read via list_drs_rules but managed in the vSphere UI. delete_drs_rule refuses non-VM-VM rules (they can carry licensing/compliance placement constraints) and records the full rule definition in both preview and result so a mistaken delete can be recreated from the audit trail.

Alarm reset blast radius: vSphere has no per-alarm clear API. reset_vcenter_alarm uses AlarmManager.ClearTriggeredAlarms, which clears all triggered alarms matching the named alarm's entity type (host/VM/all) and current status (red/yellow) — not just the one named. A bare MCP call previews every alarm it would clear (blast_radius.will_clear); confirm=True clears and scope reports it. A typo fails fast, clearing nothing.

CLI Quick Reference

# VM operations
vmware-aiops vm power-on <name> [--target <t>]
vmware-aiops vm power-off <name> [--force]
vmware-aiops vm create <name> --cpu 4 --memory 8192 --disk 100
vmware-aiops vm delete <name>
vmware-aiops vm clone <name> --new-name <new> [--to-host <host>] [--to-datastore <ds>] [--power-on]
vmware-aiops vm migrate <name> --to-host <host> [--to-datastore <ds>]
vmware-aiops vm snapshot-create <name> --name <snap> [--description <text>] [--memory]
vmware-aiops vm snapshot-list <name>
vmware-aiops vm snapshot-revert <name> --name <snap>
vmware-aiops vm snapshot-delete <name> --name <snap> [--remove-children] [--no-wait]
vmware-aiops vm task-status <task-id>                      # poll an async (--no-wait) operation by id
vmware-aiops vm set-ttl <name> --minutes 480 [--dry-run]   # double confirm; daemon auto-deletes VM on expiry

# Guest operations (requires VMware Tools)
vmware-aiops vm guest-exec <name> --cmd <script-path> --args "<args>" --user <username>
vmware-aiops vm guest-upload <name> --local ./script.sh --guest /tmp/script.sh --user <username>

# Deploy
vmware-aiops deploy ova <path> --name <vm> --datastore <ds>
vmware-aiops deploy linked-clone --source <vm> --snapshot <snap> --name <new>

# Cluster
vmware-aiops cluster create <name> --ha --drs
vmware-aiops cluster info <name>
vmware-aiops cluster drs-rules <name>                                     # list DRS rules
vmware-aiops cluster drs-rule-set <name> --rule <r> --enable|--disable [--dry-run]
vmware-aiops cluster drs-rule-create <name> --rule <r> --type antiAffinity --vm <vm1> --vm <vm2> [--disabled] [--dry-run]
vmware-aiops cluster drs-rule-delete <name> --rule <r> [--dry-run]        # VM-VM only; double confirm

# Datastore
vmware-aiops datastore browse <ds> --pattern "*.ova"

# Alarm management
vmware-aiops alarm list [--target <t>]
vmware-aiops alarm acknowledge <entity_name> <alarm_name> [--target <t>]
vmware-aiops alarm reset <entity_name> <alarm_name> [--target <t>]   # double confirm; see blast radius above

# Family
vmware-aiops hub status        # show installed family members + install commands

Full CLI reference: see references/cli-reference.md

Troubleshooting

"VM not found" error

VM names are case-sensitive in vSphere. Use exact name from vmware-monitor inventory vms.

Guest exec returns empty output

Use run_vm_guest_command_with_output instead of run_vm_guest_command — it auto-captures stdout/stderr. Basic run_vm_guest_command only returns exit code.

Deploy OVA times out

Large OVA files (>10GB) may exceed the default 120s timeout. The upload happens via HTTP NFC lease — ensure network between the machine running vmware-aiops and ESXi is stable.

Snapshot delete is slow / "still running after Ns"

Deleting an old or large snapshot consolidates its delta disk into the parent — the slowest write operation, often several minutes. vm snapshot-delete waits up to 30 min by default; if it still returns a "still running, NOT failed" message with a task id, the delete did not fail — poll it with vm task-status <task-id>. Do not re-issue the delete or hand-roll polling. For very large snapshots, prefer vm snapshot-delete <name> --name <snap> --no-wait to get the task id immediately and poll.

Plan apply fails mid-way

Run vmware-aiops plan list to see failed plan status. Ask user if they want to rollback with rollback_vm_plan. Irreversible steps (delete_vm) are skipped during rollback.

Connection refused / SSL error

1. Verify target is reachable: vmware-aiops doctor
2. For self-signed certs: set verify_ssl: false in config.yaml (lab environments only)

Setup

uv tool install vmware-aiops==2.1.0
mkdir -p ~/.vmware-aiops
vmware-aiops init  # generates config.yaml and .env templates
chmod 600 ~/.vmware-aiops/.env

Full setup guide, security details, and AI platform compatibility: see references/setup-guide.md

License

MIT — github.com/vmware-skills/VMware-AIops