TL;DR
When vSphere Supervisor deployment stalls at Installed and Started Kubernetes Node Agent on the ESXi Host, do not treat the message itself as the root cause. For the failure documented in Broadcom KB 393663, the important evidence is repeated Apply Solution activity plus vCenter WCP log entries stating that the solution specification in the desired image is incompatible with one or more ESXi hosts.
The supported remediation is to restore vSphere Lifecycle Manager compliance. Validate the cluster image, correct the ESXi base image and OEM vendor add-on as required, investigate incompatible components, verify that hosts are not booted with legacy BIOS, and remediate the hosts through vLCM. Once the hosts are compliant, vLCM can install the Spherelet VIB and Supervisor configuration can continue.
The same visible Supervisor stage can also fail because of content-library catalog problems, unreachable software depots, or ESXi DNS failures. Match the logs before applying a fix.
Introduction
A Supervisor deployment can look deceptively close to completion when it reaches the host configuration stage. The control-plane workflow has started, the cluster appears in Workload Management, and the interface reports that the Kubernetes node agent is being installed and started on the ESXi hosts.
Then nothing moves.
The Host Config Status remains Configuring, the same task stays pending, and repeated Apply Solution tasks appear in vCenter. Retrying the deployment does not change the result because the orchestration layer is waiting for a host-level prerequisite that vSphere Lifecycle Manager cannot satisfy.
Broadcom KB 393663 documents one specific version of this failure. The Spherelet VIB, which provides the Kubernetes node-agent function on each participating ESXi host, cannot be installed because the hosts are noncompliant with or incompatible with the vLCM desired image. Broadcom also identifies legacy BIOS boot mode as a condition that can leave hosts noncompliant and block the workflow.
The operational challenge is that the UI message is not unique to this root cause. Several different Spherelet delivery failures can stop at the same stage. A useful runbook must therefore separate symptom confirmation from remediation, preserve evidence, and avoid bypassing lifecycle controls before the actual failure domain is known.
Scenario and Scope
This runbook is written for administrators troubleshooting an initial vSphere Supervisor deployment with the following characteristics:
- vCenter Server 8.x
- ESXi 8.x hosts
- vSphere Supervisor or vSphere Kubernetes Service
- ESXi hosts managed with a vSphere Lifecycle Manager image
- Host Config Status stuck at Configuring
- The pending step reads Installed and Started Kubernetes Node Agent on the ESXi Host
The primary path follows KB 393663. Additional diagnostic branches are included because the same visible stage can be produced by other Spherelet, catalog, depot, or name-resolution failures.
This is not a generic Kubernetes troubleshooting article. The blocked component is the vSphere host lifecycle workflow that must place the Spherelet VIB onto each ESXi host and bring the node agent into the desired state.
Why the Deployment Stops at This Stage
Supervisor enablement crosses several control boundaries. vCenter coordinates the workflow, Workload Control Plane tracks Supervisor state, vSphere Lifecycle Manager evaluates and applies the host image, and each ESXi host must accept the Spherelet VIB.
The key dependency is easy to miss:
What to notice is that Supervisor is not directly copying a standalone agent onto an otherwise unmanaged host. The installation is reconciled through the host lifecycle system. If vLCM cannot prove that the desired image is valid for the hosts, the Spherelet installation is blocked with it.
That is why repeated retries are usually ineffective. The control plane is reissuing work against an unresolved lifecycle condition.
Symptoms That Match KB 393663
Do not classify the incident from the pending task alone. Confirm the complete signature.
vSphere Client Symptoms
The strongest UI indicators are:
- Supervisor deployment is stuck at Configuring Host Nodes.
- Host Config Status remains Configuring.
- The pending task references installation and startup of the Kubernetes node agent.
- A related message may report that the Kubernetes node agent stopped for maintenance mode.
- Repeated Apply Solution tasks appear in recent tasks.
- Cluster image compliance reports one or more hosts as noncompliant or incompatible.
WCP Log Signature
On the vCenter Server Appliance, review the Workload Control Plane log:
grep -Ei "Solution specification|incompatible with hosts|spherelet|vLCM|PMan" \ /var/log/vmware/wcp/wcpsvc.log | tail -n 200
For the KB 393663 path, look for an error stating that the solution specification in the image is incompatible with named ESXi hosts. The surrounding operation identifier commonly includes a vLCM host-add or apply operation.
The exact timestamp, host identifiers, cluster identifiers, and attempt numbers will differ. The important part is the combination of Apply Solution, incompatible hosts, and the affected host list.
vSphere Lifecycle Manager Evidence
In the vSphere Client, navigate to the affected cluster and open:
- Updates
- Hosts
- Image
- Check Compliance
Capture the result before making changes. The compliance details should identify whether the conflict is tied to the base ESXi image, vendor add-on, firmware and driver add-on, additional component, solution component, hardware warning, or another image constraint.
Diagnostic Decision Tree
The visible message identifies the stalled stage. The logs identify the actual failure domain.
This distinction prevents a common troubleshooting mistake: using a valid fix from the wrong Spherelet failure scenario.
Prerequisites and Safety Checks
Before remediation, establish the operational boundaries.
Required Access
You need:
- vCenter administrative access
- permission to inspect and edit the cluster vLCM image
- root access to the vCenter Server Appliance for log review
- temporary ESXi Shell or SSH access for read-only host checks
- out-of-band hardware management access if firmware mode may change
- access to the hardware vendor support matrix and approved OEM image information
- an approved maintenance window for host remediation or reboot
Environment Checks
Confirm the following before changing the desired image:
- DRS and evacuation capacity are sufficient for maintenance mode.
- vSAN health and resynchronization state are understood if the cluster uses vSAN.
- NSX host transport-node health is stable if NSX is present.
- Backups and recovery procedures are current for vCenter and critical management components.
- The current desired image is documented, including base image, vendor add-on, firmware add-on, and extra components.
- No unrelated lifecycle operation is already running.
Change-Control Warning
A vLCM image edit is not a cosmetic change. It can alter drivers, firmware coordination, host components, and reboot requirements across the cluster.
Do not remove an OEM add-on or third-party component merely to make the compliance screen turn green. Identify the owner, support requirement, and replacement path first.
Runbook Stage: Confirm the Exact Failure Signature
Start with the lowest-risk evidence collection.
Capture the Supervisor State
Record:
- Supervisor name
- cluster name
- affected ESXi hosts
- Host Config Status
- exact pending task text
- first observed failure time
- recent deployment or upgrade changes
- screenshots of repeated Apply Solution tasks
Review WCP and vLCM Logs
On the vCenter Server Appliance, collect the relevant WCP lines:
grep -Ei "Apply Solution|Solution specification|incompatible with hosts|spherelet|PMan|vLCM" \ /var/log/vmware/wcp/wcpsvc.log | tail -n 300
If the WCP log points to an Apply Solution failure, correlate it with the Update Manager log:
grep -Ei "ClusterApplySolutionTask|VibDownloadError|incompatible|spherelet" \ /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log | tail -n 300
Successful confirmation for the KB 393663 path means you can answer yes to all three questions:
- Is Supervisor stalled at the Kubernetes node-agent stage?
- Is vLCM reporting host noncompliance or incompatibility?
- Do the WCP or Update Manager logs tie the failure to Apply Solution or the desired image?
If not, stop and use the differential diagnosis section later in this article.
Runbook Stage: Capture Host and Image State
Collect a baseline from every ESXi host before remediation. The commands below are read-only.
# Record the ESXi release and build vmware -vl # Record the installed image profile esxcli software profile get # Check whether the Spherelet VIB is already installed esxcli software vib list | grep -i spherelet # Check whether the host presents UEFI or legacy BIOS firmware vsish -e get /hardware/firmwareType
For environments where normal management agents are impaired, Broadcom examples sometimes use localcli to list VIBs. Keep any localcli use read-only unless Broadcom Support explicitly directs otherwise. Changing host state with localcli bypasses normal management coordination and can create a second problem while you are investigating the first.
Create a small evidence table for the change record:
| Host | ESXi build | Image profile | Spherelet present | Firmware type | vLCM status |
|---|---|---|---|---|---|
| esx01 | Record value | Record value | Yes or no | UEFI or Legacy BIOS | Compliant, noncompliant, or incompatible |
| esx02 | Record value | Record value | Yes or no | UEFI or Legacy BIOS | Compliant, noncompliant, or incompatible |
| esx03 | Record value | Record value | Yes or no | UEFI or Legacy BIOS | Compliant, noncompliant, or incompatible |
This table often reveals whether the problem is cluster-wide or isolated to one host with a different firmware mode, OEM image, driver set, or build.
Runbook Stage: Interpret the vLCM Compliance Failure
The word noncompliant is not a diagnosis. Expand the details until the conflicting element is known.
Base ESXi Image Mismatch
The desired cluster image may specify a different ESXi release or patch level than one or more hosts can accept. Validate:
- the exact target ESXi version and build
- server hardware compatibility
- vCenter and ESXi interoperability
- Supervisor support for the selected release
- boot-device capacity and firmware requirements
Do not select a newer base image solely because it appears in the depot. The target must be supported by the server platform and the surrounding VCF or vSphere stack.
OEM Vendor Add-On Mismatch
If the hosts were installed from an OEM-customized image, the desired image should usually include the correct vendor add-on for the server model and target ESXi release.
A missing, stale, or incompatible vendor add-on can create driver and provider differences that prevent compliance. Verify the add-on against the hardware vendor’s current support guidance rather than copying the configuration from another cluster with different servers.
Additional VIBs, async drivers, hardware-management providers, NSX components, storage plug-ins, or partner agents can make the desired image incompatible.
For each conflicting component, determine:
- which product installed it
- whether it is still required
- which team owns it
- whether a compatible version exists
- whether the component must be upgraded before host remediation
- whether removal is supported and reversible
The goal is not to produce a minimal image. The goal is to produce a supported image that represents the actual platform dependencies.
Depot or Repository Availability
A host can appear incompatible when vLCM cannot complete mandatory validation because the configured online or offline depot is unavailable. In that case, the image definition may be correct, but the lifecycle service cannot retrieve or validate its components.
If Update Manager logs show VibDownloadError, repository reachability, certificate trust, proxy configuration, or depot configuration becomes the first repair path. Do not rewrite the desired image to work around an unavailable source.
Runbook Stage: Resolve Legacy BIOS Boot Mode
KB 393663 specifically identifies legacy BIOS boot as a condition that can leave ESXi hosts noncompliant and prevent Spherelet installation.
Use the firmware check on every host:
vsish -e get /hardware/firmwareType
If the output reports Legacy BIOS, treat the correction as a hardware and boot-path change, not as a quick Supervisor setting.
Plan the Transition Before Changing Firmware
At minimum, document:
- current boot device and boot order
- disk partitioning and bootbank health
- server model and firmware baseline
- storage adapter and network adapter PCI identities
- active vmnic names, MAC addresses, and uplink mappings
- out-of-band console access
- vendor support for UEFI on the exact server generation
- recovery path if ESXi does not boot or networking changes
Switching firmware mode can affect how the platform presents boot devices and PCI hardware. A safe change includes console access, a host evacuation plan, post-change network validation, and a tested recovery procedure.
Do not toggle legacy BIOS to UEFI across every host at once. Use a pilot host, validate boot, storage, networking, vCenter connectivity, and vLCM compliance, then continue through the cluster according to the approved maintenance plan.
Runbook Stage: Correct the Desired Cluster Image
In the vSphere Client:
- Select the affected cluster.
- Open Updates.
- Select Hosts, then Image.
- Review the current desired image.
- Select the supported ESXi base version.
- Add or correct the OEM vendor add-on when required.
- Validate firmware and driver add-ons if a hardware support manager is integrated.
- Review additional components and solution dependencies.
- Save the image.
- Run Check Compliance again.
A useful review sequence is:
Do not proceed to remediation while the reason for incompatibility remains unexplained. A clean precheck is the change gate, not an optional report.
Once the desired image is correct and the compliance analysis is understood, remediate through the supported vLCM workflow.
Use Controlled Host Sequencing
Where cluster capacity and the platform support model permit it:
- remediate one host first
- confirm maintenance mode entry and workload evacuation
- monitor image staging and installation
- reboot only when required
- validate management, storage, network, and cluster health
- confirm the host returns compliant
- then continue with the remaining hosts
For vSAN clusters, select an evacuation mode that matches the maintenance duration and available capacity. For NSX-backed clusters, verify transport-node and tunnel health after each host returns. For VCF-managed environments, ensure the action remains consistent with the supported VCF lifecycle path and does not create undocumented out-of-band drift.
Do Not Force Past Hardware Warnings
A command-line option that suppresses a warning does not resolve the underlying support condition. Legacy BIOS, unsupported hardware, incompatible drivers, or insufficient boot media should be corrected or explicitly accepted through the appropriate vendor and support process.
The objective is a compliant, supportable host image, not merely a completed task.
Validation should prove both lifecycle recovery and Supervisor recovery.
Validate vLCM State
Confirm that:
- every participating host is compliant with the desired image
- no host remains incompatible
- no Apply Solution task is repeating
- recent Update Manager logs contain no new image or VIB installation failures
Validate Spherelet Installation
On each ESXi host:
esxcli software vib list | grep -i spherelet
Successful output should show a Spherelet package entry. Record the installed version for all hosts and confirm the versions are consistent with the target Supervisor and host image.
Validate Supervisor Host Configuration
In Workload Management, confirm that:
- Host Config Status leaves Configuring
- each host becomes ready for Supervisor use
- the Kubernetes node-agent task completes
- the overall Supervisor state is healthy
- no new host configuration error appears
Recheck WCP Logs
Review recent WCP activity after remediation:
grep -Ei "spherelet|Apply Solution|incompatible with hosts|error wcp" \ /var/log/vmware/wcp/wcpsvc.log | tail -n 200
The absence of the old incompatibility error is necessary, but it is not sufficient by itself. The operational pass condition is successful Spherelet presence plus a healthy Supervisor host state.
When the Same Message Is Not KB 393663
The UI status is a location in the workflow, not a unique error code. Use the evidence below to avoid applying the wrong repair.
| Evidence | More likely failure domain | Recommended next action |
|---|---|---|
Solution specification in the image are incompatible with hosts plus vLCM noncompliance |
KB 393663 image or firmware path | Correct desired image, resolve firmware issue, and remediate through vLCM |
catalog is not ready, supervisor content is being processed, or malformed Supervisor OVF template |
Invalid or unhealthy content library | Repair or unassign the affected Supervisor content library according to the matching Broadcom KB |
VibDownloadError for an online or custom depot component |
vLCM depot, proxy, certificate, or repository reachability | Restore depot access, rerun compliance, then allow orchestration to resume |
Spherelet VIB download reports Name or service not known |
ESXi DNS or name-resolution failure | Correct ESXi DNS configuration and verify resolution of the vCenter artifact endpoint |
| Spherelet is absent, but none of the documented signatures appear | Unclassified host configuration failure | Preserve WCP, Update Manager, and ESXi logs; engage Broadcom Support before destructive recovery |
A manual Spherelet VIB copy and install appears in separate Broadcom troubleshooting articles for other signatures and versions. That does not make it the preferred response to KB 393663. If vLCM reports the desired image as incompatible, bypassing vLCM can create image drift and leave the underlying lifecycle problem unresolved.
Common Mistakes That Make the Incident Harder
Reinstalling ESXi Before Reading the Logs
A host rebuild can remove the visible symptom while destroying the evidence that explains why the lifecycle workflow failed. It can also reintroduce the same problem if the rebuilt host uses the same incompatible image or legacy BIOS mode.
Deleting and Recreating the Supervisor Immediately
Recreating Supervisor repeats the workflow against unchanged hosts. Collect the WCP operation identifiers, vLCM compliance report, and host state first.
Manually Installing Spherelet as the First Fix
Manual installation can be valid only when the matching Broadcom procedure supports it for the exact failure. It should not replace correction of vLCM noncompliance.
Removing OEM or Third-Party Components Without Ownership Review
A component that blocks compliance may still provide a required NIC, storage, hardware-management, security, or NSX function. Removing it without a dependency review can trade a deployment failure for a host outage.
Changing BIOS Mode Without Console and Recovery Planning
Firmware changes can affect the boot path and device presentation. Out-of-band access and a pilot-host validation are mandatory operational controls.
Treating One Compliant Host as Proof of Cluster Readiness
Supervisor depends on every participating host reaching the required state. Compare image, build, firmware, and Spherelet evidence across the complete cluster.
Rollback and Fallback Guidance
Rollback depends on how far the change progressed.
If the Desired Image Was Edited but Not Remediated
Restore the previously documented desired image components, save the image, and rerun compliance. Because host state has not changed, this is the lowest-risk rollback point.
If a Pilot Host Was Remediated
Validate whether vLCM offers a supported rollback for the applied image and whether the previous image remains compatible with vCenter, Supervisor, hardware, and other cluster solutions. Do not assume that a host-level software rollback is always available or safe.
If rollback is not supported, the recovery path may require a vendor-approved image, host reinstallation, or Broadcom Support guidance. Preserve the remediation logs before taking that path.
If Firmware Mode Was Changed
Use out-of-band console access to evaluate boot, storage, and network state. Reverting firmware mode may be part of recovery, but only after confirming the boot-device implications and vendor guidance. Do not alternate repeatedly between legacy BIOS and UEFI without documenting each state.
If Supervisor Remains Stuck After Compliance Is Restored
Do not immediately repeat all remediation. Collect fresh evidence and compare the new error with the original one. The lifecycle incompatibility may be fixed while a second catalog, DNS, certificate, or connectivity issue is now visible.
Evidence to Collect Before Escalation
A strong support bundle shortens the path from generic symptom to actionable diagnosis.
Collect:
- Supervisor name and deployment timestamp
- affected cluster and host names
- screenshots of Host Config Status and pending task details
- recent task history showing repeated Apply Solution attempts
- relevant
wcpsvc.logexcerpts with operation identifiers - relevant Update Manager log excerpts
- vLCM desired image definition
- vLCM compliance report for each host
- ESXi version and build from each host
- installed image profile from each host
- Spherelet VIB presence and version
- firmware type from each host
- recent changes to ESXi images, vendor add-ons, firmware, DNS, proxy, content libraries, or depots
- confirmation of hardware compatibility and OEM support
- vCenter and ESXi support bundles when requested
The best escalation question is not, “Why is Supervisor stuck?” It is, “Why does vLCM consider this exact desired image incompatible with these exact hosts, and what supported change restores Spherelet installation?”
Conclusion
The pending message Installed and Started Kubernetes Node Agent on the ESXi Host is not the diagnosis. It is the point where Supervisor enablement is waiting for host lifecycle reconciliation to finish.
For the condition documented in KB 393663, the decisive evidence is vLCM noncompliance or incompatibility combined with Apply Solution failures in the WCP workflow. The supported response is to correct the desired cluster image, resolve legacy BIOS where applicable, and remediate the hosts through vSphere Lifecycle Manager. Once the hosts are compliant, the Spherelet VIB can be installed and Supervisor configuration can continue.
The operational discipline matters as much as the technical fix. Preserve the logs, validate every host, treat firmware conversion as a planned maintenance activity, and do not bypass vLCM merely because a manual VIB command exists in a different KB.
Fix the lifecycle state first. Then prove that Spherelet is installed, the host configuration is ready, and Supervisor has returned to a healthy operating state.
