The vCenter Log Partition Runbook: Find Growth, Preserve Evidence, Restore Headroom

A full /storage/log partition on a vCenter Server Appliance is not just a housekeeping problem. It is a management-plane risk.

In a standalone vSphere environment, it can interrupt administration, log collection, patching, and service stability. In VMware Cloud Foundation, the blast radius is larger because vCenter is tied into SDDC Manager workflows, workload domain lifecycle operations, and operational visibility.

Broadcom KB313077 calls out /storage/log exhaustion directly and warns that deleting critical files can prevent the vCenter Server Appliance from working, while resizing appliance disks carries data corruption risk if handled poorly.

The wrong response is to SSH into the appliance and start deleting large files until the alarm clears.

The right response is a runbook: confirm the partition, preserve enough evidence to understand the cause, collect support data without making the partition worse, restore safe headroom, and then prevent the same pattern from returning.

This article walks through that workflow.

What This Runbook Helps You Accomplish

By the end of this runbook, you should be able to:

• Confirm whether the issue is actually /storage/log.
• Identify the largest files and highest-growth directories.
• Preserve triage evidence before cleanup.
• Collect support bundles safely when /storage/log is already constrained.
• Remove low-risk temporary files such as downloaded support bundles after validation.
• Avoid deleting active logs, database files, or appliance-critical content blindly.
• Decide when cleanup is enough and when the log partition needs resizing.
• Add monitoring so the next incident is caught before vCenter services are at risk.

The article assumes vCenter Server Appliance, not the retired Windows-based vCenter deployment model. Broadcom’s current log-location reference notes that vCenter Server Appliance logs are organized under /var/log/vmware/, while this runbook focuses on the appliance log partition mounted at /storage/log.

Scenario: The Alarm Is Already Firing

The common alert path looks familiar:

• VAMI shows /storage/log at high utilization.
• vSphere Client reports Log Disk Exhaustion.
• Log bundle export fails or produces an incomplete bundle.
• vCenter services are unstable or unavailable.
• In worse cases, the vSphere Client returns 503 Service Unavailable.

Broadcom KB 313077 lists the Log disk exhaustion alarm, VAMI /storage/log warnings, possible 503 Service Unavailable errors, and issues viewing some vSAN-related options when the log partition is under pressure. It also documents VAMI warning behavior at 75% continuous usage and red critical alarms at 85%.

There is also a broader vCenter disk-space behavior to keep in mind: Broadcom’s VCSA disk-space KB says vCenter disk warnings commonly begin around 80%, and when vCenter file systems reach 95%, vmware-vpxd may be turned off automatically to help protect the database from corruption.

That is why the runbook should not start with cleanup.

It starts with containment and evidence.

Why /storage/log Matters in VCF Operations

In VMware Cloud Foundation, vCenter is not an isolated management appliance. It participates in lifecycle workflows, upgrade prechecks, health checks, and domain operations.

Broadcom documents a VCF case where a vCenter upgrade triggered from SDDC Manager failed because /storage/log was full before or during the upgrade. The remediation path included freeing space in /storage/log and retrying the SDDC Manager workflow.

That makes /storage/log capacity a lifecycle readiness item, not just a reactive alarm.

A practical operating model should treat log partition headroom as part of:

• vCenter health.
• VCF upgrade readiness.
• Supportability.
• Evidence preservation.
• Monitoring and alert response.
• Change-window preparation.

If the first time the team checks /storage/log is during a failed upgrade or an active support case, the operational process is already behind.

The Runbook at a Glance

The diagram below shows the flow. The key point is the separation between discovery, evidence preservation, cleanup, and prevention. Those steps are often collapsed during an outage, which is how useful evidence gets deleted before anyone understands the root cause.

Prerequisites and Safety Checks

Before touching files, confirm the operational state.

You need:

• Root or approved administrative access to the vCenter Server Appliance.
• Console access or SSH access.
• A recent, restorable vCenter backup.
• Awareness of Enhanced Linked Mode or VCF domain dependencies.
• A support case reference if this is already tied to a Broadcom SR.
• Enough free space on another appliance partition or external destination if you need to preserve evidence or generate a support bundle.

Broadcom KB 313077 explicitly warns to ensure a good VAMI file backup, VADP backup, or both before deleting files or resizing VCSA disks. If resizing becomes necessary, Broadcom’s disk-space guidance also warns that a good backup is critical before resizing appliance disks.

A few practical guardrails:

Do not delete files from /storage/db, /storage/seat, /storage/core, or / just because they appear in a generic disk report. This article is scoped to /storage/log.

Do not delete active log files without preserving evidence. Removing an active file can also fail to reclaim space until the process releases the file handle.

Do not treat /storage/archive the same way as /storage/log. Broadcom’s VCSA disk-space KB notes that /storage/archive being high or full can be expected by design in vCenter 6.7 and later.

Do not resize first unless you have already determined the partition is genuinely undersized or you need emergency headroom. Resizing can hide a runaway logging condition without fixing the cause.

Stage 1: Confirm the Partition and Current Risk

Start with read-only checks.

# If you land in the appliance shell, enable and enter Bash.
shell.set --enabled True
shell

# Confirm filesystem usage.
df -h

# Show only filesystems at or above the investigation threshold.
df -h | awk '0+$5 >= 78 {print}'

# Confirm inode usage as well.
df -ih

# Focus directly on the log partition.
df -hT /storage/log
df -ih /storage/log

Broadcom’s VCSA disk-space KB uses df -h and an awk filter around 78% to catch partitions that may temporarily cross warning thresholds and then fall back below the visible alarm threshold.

What you are looking for:

Check	Good Sign	Risk Sign
/storage/log space	Below warning threshold with available GB	75–85% and rising
/storage/log inodes	Plenty of free inodes	High inode use with many small files
Other partitions	Normal usage	/storage/db, /storage/seat, /, or /storage/core also high
vCenter services	Running	vmware-vpxd stopped or client unavailable
Timing	Stable usage	Rapid growth over minutes

If multiple partitions are high, do not use a /storage/log runbook as a universal cleanup procedure. Identify each partition separately.

Stage 2: Preserve Evidence Before Cleanup

A full log partition is often a symptom, not the root cause. It may be caused by support bundles left behind, unusually high event volume, a service repeatedly writing errors, failed cleanup behavior, or a version-specific known issue. Broadcom KB 313077 lists potential causes including log bundles not being cleared, high-frequency events, services failing to clean up files, and an undersized /storage/log partition.

Before deleting anything, capture a small triage evidence package.

Use a destination that has enough space. The example below uses /storage/core, but only use it after confirming it has headroom.

# Confirm there is space before using /storage/core.
df -h /storage/core

RUNID="$(date +%Y%m%d-%H%M%S)"
EVDIR="/storage/core/vc-log-triage-${RUNID}"

mkdir -p "$EVDIR"

hostname -f > "$EVDIR/hostname.txt"
date -Is > "$EVDIR/date.txt"

df -hT > "$EVDIR/df-hT.txt"
df -ih > "$EVDIR/df-ih.txt"

du -xhd1 /storage/log 2>/dev/null | sort -hr > "$EVDIR/storage-log-topdirs.txt"

find /storage/log -xdev -type f \
  -printf '%s %TY-%Tm-%Td %TH:%TM %p\n' 2>/dev/null \
  | sort -nr \
  | head -100 > "$EVDIR/storage-log-topfiles.txt"

find /storage/log -xdev -type f -size +100M \
  -exec ls -lh {} \; 2>/dev/null \
  > "$EVDIR/storage-log-files-over-100M.txt"

tar -czf "${EVDIR}.tgz" -C "$(dirname "$EVDIR")" "$(basename "$EVDIR")"

ls -lh "${EVDIR}.tgz"

This does not preserve every log file. It preserves enough state to explain what was consuming space before remediation started.

That distinction matters. In a production incident, you often need to restore headroom quickly, but you still want a record of what was large, where it lived, when it was modified, and which partition was actually full.

Also remember that support bundles can include sensitive operational data. Broadcom notes that support bundle collection can include logs, configuration files, product-specific data, and core dump material depending on the product and collection method, so these files should be handled according to corporate policy.

Stage 3: Find What Is Consuming /storage/log

Now move from partition-level confirmation to file-level discovery.

Broadcom KB 313077 recommends running a largest-file search from /storage/log using find, du, sort, and head. The broader VCSA disk-space KB also recommends du, find for files larger than 100 MB, and a directory file-count check when many small files are the issue.

Use these commands as the working set:

cd /storage/log

# Top large files.
find . -type f -print0 | xargs -0 du -h | sort -rh | head -n 20

# Top directories on the same filesystem.
du -xhd1 /storage/log 2>/dev/null | sort -hr | head -n 20

# Files larger than 100 MB.
find /storage/log -xdev -type f -size +100M -exec ls -lh {} \; 2>/dev/null

# Directories with high file counts.
find /storage/log -xdev -type d -exec sh -c '
  count=$(find "$1" -maxdepth 1 -type f 2>/dev/null | wc -l)
  if [ "$count" -gt 100 ]; then
    echo "$count $1"
  fi
' sh {} \; | sort -nr | head -n 30

You are trying to classify the pattern, not just identify the biggest number.

Common patterns look like this:

Pattern	What It Usually Means	First Response
Large *.tgz files	Support bundles or exported diagnostics left behind	Confirm downloaded, then remove selected bundle files
One huge *.log, *.stdout, or *.stderr	Active service logging repeatedly	Preserve evidence, identify service, check known KBs
Thousands of small files	Rotation, cleanup, or service behavior issue	Count files, identify directory, check version-specific KBs
Growth during log export	Support bundle generation is consuming /storage/log	Generate bundle elsewhere using vc-support -w
Normal logs but tiny partition	Partition undersized for current version or workload	Consider supported resize after backup

Do not stop at “file is large.” Ask why it is large.

Stage 4: Map the Culprit to a Known Issue or Behavior

Broadcom KB 313077 includes a known-issue matrix for /storage/log growth across vCenter versions and file patterns. The KB points to examples involving SSO logs, excessive pod-startup logs, analytics logs, EAM web logs, SPS runtime logs, support bundles, Java heap dumps, and WCP stream logs.

Your triage output should give you three useful data points:

1. The exact file path.
2. The vCenter version and build.
3. Whether the file is active growth, stale artifact, or many-file accumulation.

Capture version information:

vpxd -v 2>/dev/null || true
cat /etc/applmgmt/appliance/software_update_state.conf 2>/dev/null || true

Then classify the culprit:

Culprit Type	Example Investigation Question	Cleanup Posture
Stale support bundle	Was this generated through vSphere Client or VAMI and already downloaded?	Usually safe to remove after confirmation
Active service log	Is a service repeatedly writing errors?	Preserve sample, address service/root cause
Known version defect	Does the file match a Broadcom KB for this build?	Follow the specific KB remediation
Undersized partition	Is normal log behavior exceeding an old disk layout?	Resize after backup and snapshot checks
Event storm	Are tasks, events, or integrations generating excessive log volume?	Fix upstream behavior, not just the file

This is where the runbook becomes more than Linux cleanup.

You are connecting disk consumption to vCenter behavior.

Stage 5: Collect Logs Without Making /storage/log Worse

Support bundle collection can make the problem worse if it writes into an already-constrained /storage/log.

Broadcom’s diagnostic collection KB documents vCenter support bundle collection through the vSphere Client, while another Broadcom KB explains that support bundles generated from the vSphere Client are created in /storage/log and can fail when that partition has high usage or low free space.

When /storage/log does not have enough space, use an alternate destination. Broadcom documents that vc-support generates bundles in /storage/log by default, but the -w option can place the bundle on another filesystem with more available space.

# Check available space before choosing a destination.
df -h

# Default behavior when /storage/log has enough headroom.
vc-support -l

# Alternate destination when /storage/log is constrained.
vc-support -w /storage/core

Use /storage/core only if it has enough free space and your operational policy allows it. If /storage/core contains large core files, review policy before deleting or moving anything.

If vSAN log collection is causing collection time or size issues, Broadcom also documents manifest-based vc-support options and examples for including or excluding specific manifests, including use of -w /storage/core with manifest exclusions.

Stage 6: Restore Headroom Safely

Once you know what is consuming space, use the least destructive remediation that restores enough headroom.

Option A: Remove Confirmed Support Bundles

Support bundles are one of the cleaner cleanup targets, but only after you confirm they are no longer needed or have been downloaded.

Broadcom documents a vCenter 8.0 issue where support bundle files exported through the vSphere Client might not be removed from /storage/log after download. The documented workaround is to manually remove the file from /storage/log after export or use VAMI cleanup after download.

Start with inspection:

find /storage/log -maxdepth 1 -type f -name "*.tgz" -exec ls -lh {} \;

Then remove only the selected file or files:

# Use exact filenames where possible.
rm -i /storage/log/.tgz

Avoid broad deletion until you understand naming and retention in your environment.

Option B: Remove Old Rotated Logs Only After Policy Review

If the partition is filled with old compressed rotated logs, confirm whether they are still needed for troubleshooting, audit, or support.

Inspect first:

find /storage/log -xdev -type f \
  \( -name "*.gz" -o -name "*.zip" -o -name "*.old" \) \
  -mtime +30 -exec ls -lh {} \;

Then remove selected files only after approval:

rm -i /path/to/selected/old-rotated-log.gz

This is not the first cleanup option during an unresolved incident. Old rotated logs may still be useful if the issue has been building over time.

Option C: Handle a Huge Active Log Carefully

If one active log is consuming the partition, do not blindly delete it.

Preserve metadata and a sample first:

FILE="/storage/log/path/to/large-active.log"
RUNID="$(date +%Y%m%d-%H%M%S)"
EVDIR="/storage/core/vc-log-active-file-${RUNID}"

mkdir -p "$EVDIR"

stat "$FILE" > "$EVDIR/file.stat.txt"
head -n 500 "$FILE" > "$EVDIR/file.head.txt"
tail -n 5000 "$FILE" > "$EVDIR/file.tail.txt"

Then identify which service owns the log, check whether the exact filename and vCenter build map to a Broadcom known issue, and remediate the cause. If you must truncate a log in an emergency, treat it as an exception and document the approval:

# Emergency-only after evidence preservation and approval.
: > "$FILE"

Truncation preserves the file path and inode, which is usually less disruptive than deleting an active file, but it still destroys log history. Use it only when the operational risk of a full partition is higher than the loss of local log content.

Option D: Resize /storage/log When It Is Truly Undersized

Sometimes cleanup is not enough. Older appliances upgraded across multiple major versions can have a log partition that is too small for current support bundle and logging behavior.

Broadcom’s KB for vCenter 8.0 states that from 8.0 U3h onward, /storage/log is 50 GB by default for fresh deployments, major upgrades from 7.x to 8.x, and patching using reduced downtime upgrade. It also recommends increasing /storage/log to 50 GB after in-place updates to 8.0 U3h and later.

Before resizing:

• Confirm you have a current backup.
• Confirm there are no VM snapshots blocking disk expansion.
• Confirm the exact virtual disk mapped to /storage/log.
• Follow the Broadcom procedure for your version.
• Do not resize the wrong VMDK.

Broadcom’s 50 GB KB calls out a full file-level backup before extending disks and removing snapshots if disk extension is unavailable.

Stage 7: Validate Recovery

After cleanup or resizing, validate from the appliance and the management interfaces.

df -hT /storage/log
df -ih /storage/log

# Check service state.
service-control --status --all

If vmware-vpxd or related services stopped because disk pressure crossed the protection threshold, do not restart services until /storage/log has meaningful headroom and the growth source is controlled.

When ready:

service-control --start vmware-vpxd

Then validate:

• VAMI health status.
• vSphere Client login.
• Recent tasks and events.
• SDDC Manager inventory and health, if this is a VCF environment.
• Any failed lifecycle workflow that may need retry.
• Whether /storage/log continues growing after cleanup.

A good recovery is not “the alarm is gone.”

A good recovery is “the alarm is gone, services are stable, support data is preserved, and the growth pattern is understood.”

Stage 8: Prevent Recurrence With Monitoring

Broadcom’s monitoring guidance for vCenter appliance disks includes monitoring disk use, configuring alarms, setting email alerts, managing statistics level, retention policy, and burst events. It also notes that alarms specific to /storage/seat and /storage/log filling were introduced in vCenter Appliance 6.7.

At minimum, monitor:

Signal	Warning	Critical	Why It Matters
/storage/log capacity	75%	85%	Aligns with VAMI behavior for log disk exhaustion
/storage/log inodes	75%	85%	Catches many-small-file failures
Growth rate	>5% per day	>10% per day	Detects runaway logging early
Support bundle count	Any stale bundle	Multiple stale bundles	Prevents artifacts from consuming logs
vCenter service health	Any stopped service	vmware-vpxd stopped	Protects management-plane availability
Upgrade readiness	Precheck warning	Lifecycle failure	Prevents VCF upgrade disruption

Here is a simple external SSH-based monitoring example. This is intentionally read-only and should run from an approved monitoring host. In production, prefer your standard monitoring platform, VMware Aria Operations / VCF Operations, SNMP, syslog, or enterprise observability tooling where available.

#!/usr/bin/env bash
set -euo pipefail

VCENTER="vcsa01.lab.local"
WARN=75
CRIT=85

usage="$(
  ssh -o BatchMode=yes root@"$VCENTER" \
  "df -P /storage/log | awk 'NR==2 {gsub(/%/,\"\",\$5); print \$5}'"
)"

if [ "$usage" -ge "$CRIT" ]; then
  echo "CRITICAL: $VCENTER /storage/log is ${usage}% used"
  exit 2
elif [ "$usage" -ge "$WARN" ]; then
  echo "WARNING: $VCENTER /storage/log is ${usage}% used"
  exit 1
else
  echo "OK: $VCENTER /storage/log is ${usage}% used"
  exit 0
fi

What to modify:

• Replace vcsa01.lab.local with the appliance FQDN.
• Use your approved authentication model.
• Set thresholds to match your operational policy.
• Send results into your monitoring system rather than relying on terminal output.

This script should not become the only control. It is a lightweight example of the signal you want your monitoring platform to track continuously.

Command Reference

Task	Command
Enter Bash from appliance shell	shell.set --enabled True then shell
Show filesystem usage	df -h
Show high-usage filesystems	df -h | awk '0+$5 >= 78 {print}'
Show inode usage	df -ih
Check /storage/log directly	df -hT /storage/log
Find largest files	find . -type f -print0 | xargs -0 du -h | sort -rh | head -n 20
Find top directories	du -xhd1 /storage/log | sort -hr | head -n 20
Find files over 100 MB	find /storage/log -xdev -type f -size +100M -exec ls -lh {} \;
Generate default support bundle	vc-support -l
Generate support bundle elsewhere	vc-support -w /storage/core
Check all service state	service-control --status --all
Start vCenter service	service-control --start vmware-vpxd

Troubleshooting Notes and Gotchas

The Largest File Is Not Always Safe to Delete

A large support bundle is different from a large active service log. A support bundle may be removable after download. An active service log may be the only local evidence of a repeated service failure.

A Successful Cleanup Does Not Prove the Issue Is Fixed

If a service is writing several GB per hour, deleting old files only buys time. Watch the growth rate after cleanup.

Support Bundle Collection Can Fail Because /storage/log Is Already Full

Broadcom documents failed or incomplete log bundle collection when /storage/log has high usage or low available space and recommends using an alternate destination with vc-support -w when needed.

VCF Lifecycle Failures May Need Workflow Retry

If /storage/log filled during a VCF-triggered vCenter upgrade, clearing space is not the whole recovery. You may need to retry the SDDC Manager workflow or follow the specific Broadcom guidance for the failed state.

Some Fixes Are Version-Specific

Do not apply a workaround for vCenter 7.0 to vCenter 8.0 just because the filename looks similar. Match the exact vCenter version, build, file path, and symptom pattern.

Final Operational Checklist

Before declaring the incident resolved, confirm:

• /storage/log has safe headroom.
• Inode usage is healthy.
• vCenter services are running.
• VAMI storage health is green or understood.
• vSphere Client login works.
• SDDC Manager health is clean, if applicable.
• Support evidence was preserved.
• Any generated support bundle was downloaded and removed if no longer needed.
• The culprit file or directory was mapped to a cause.
• Monitoring exists for capacity and growth rate.
• A follow-up patch, resize, or configuration action is tracked.

Conclusion

The temptation with /storage/log exhaustion is to treat it as a disk cleanup task. That is too narrow.

A full vCenter log partition can affect supportability, lifecycle operations, and management-plane availability. In VCF, it can also interfere with SDDC Manager-driven workflows. The better pattern is disciplined and repeatable: confirm the partition, preserve evidence, identify the growth source, collect logs safely, restore headroom, validate services, and monitor for recurrence.

The goal is not just to make the alarm disappear.

The goal is to leave vCenter healthier than it was before the alert fired.

External Links and Sources