Showing Posts From

Ucs

vCenter 8 and ESXi 8 — Preupgrade Issues to be fixed

Follow-up to VMware. Upgrading vCenter and ESXi is rarely “click Next until done.” The preflight(pre-upgrade) wizard and esxcli output tell you most of what will bite you — if you read them before the change window. Page: VMware. Physical layer: Physical Hardware. vCenter 7 → 8: errors that stop the upgrade These are from real preflight runs. The categories repeat across environments. Host profiles older than 6.7 (hard stop) vCenter 8 rejects source environments with host profile versions below 6.7. Preflight lists each non-compliant profile by name — cluster baseline profiles, one-off “reset root password” profiles, lab templates, all of it. Fix before vCenter: upgrade or recreate those profiles to 6.7+ (VMware KB 52932). Do not assume “we barely use host profiles” — if one stale profile exists, the vCenter upgrade stops. Host profile work belongs before vCenter and before you finish upgrading ESXi hosts still on very old builds. Extensions that may not survive the move Aria Operations / legacy vROps plugins often show as extensions that cannot be upgraded automatically or may not work on the new vCenter. Plan to verify compatibility, re-register after cutover, and if monitoring is managed by different team have them notified — not after having false alerts in next morning. Lifecycle Manager baggage that will not copy Guest OS patch baselines, old host upgrade baselines, and ESXi 6.5-and-lower patch content do not migrate to the new Lifecycle Manager. That is not a bug — rebuild baselines on the new vCenter rather than hunting for “missing” files after upgrade. DRS mode during host work Preflight warns when clusters are Fully Automated DRS while hosts tied to the upgrade are still in play. For the duration of host patching or vCenter migration windows, I move affected clusters to manual or partially automated DRS so VMs are not relocated mid-maintenance. vCenter appliance sizing and NICs Common warnings that are easy to dismiss:Source vCenter has more vCPUs than the default target appliance — resize after stage 2 if you need parity. Multiple NICs on the source — only active adapters come across cleanly; secondary NICs may need manual re-add post-upgrade.When stage 1 dies on database export I have hit “Internal error occurs during Export of VMware vCenter Server Database” mid-upgrade. That is support-bundle territory — not something to retry blindly three times in the same window. Take a snapshot or backup of the source appliance before you start, and have a rollback plan ready. ESXi 8 on UCS and rack servers: bundles beat generic zip On Cisco UCS hosts I use the vendor-customized ESXi depot (Cisco UCS add-on profile), not the generic VMware zip alone. On Dell rack servers, the Dell customized bundle. Standard profiles often fail with driver/VIB dependency errors on hardware that has been running fine for years. Pattern on the host Enter maintenance mode first. Copy the bundle to local VMFS (not HTTP — offline install rejects remote URIs): esxcli system maintenanceMode set --enable true# List profiles in the depot esxcli software sources profile list -d /vmfs/volumes/<DATASTORE>/ISO/<vendor-or-standard-depot.zip># Apply the matching profile name from that list esxcli software profile update \ -d /vmfs/volumes/<DATASTORE>/ISO/<vendor-or-standard-depot.zip> \ -p <Profile-Name-From-List> \ --no-hardware-warning--no-hardware-warning is a conscious choice — only after the profile matches your hardware support matrix, not because the patch is late. If standard depot fails with dependency errors on UCS or blade hardware, switch to the OEM/custom bundle before chasing individual VIBs. ESXi 7.x patch lesson that still applies at 8 On a 7.0.2 → 7.0.3 patch, DependencyError from orphaned QLogic FC VIBs (qedi / qedf) still installed while the active HBAs were Cisco VIC (nfnic) blocked the depot. The adapters in use did not need those VIBs — removing the stale packages and rebooting cleared it: esxcli software vib list | grep -E 'qedi|qedf' esxcli storage core adapter list # confirm nothing active uses them esxcli software vib remove -n qedf esxcli software vib remove -n qedi reboot # then retry profile update with the correct bundleLesson: installed ≠ in use. Before ESXi 8, inventory VIBs against actual vmhba drivers, not against what was true when the host was built. Offline bundle must be local esxcli software vib install -d http://... fails with “Only server local file path is supported.” Copy the depot to VMFS (wget to a datastore path works), then point esxcli at the local zip. Pacing host upgrades in big clusters For multi-host UCS clusters I batch 4–8 hosts per maintenance window, maintenance mode → evacuate → patch → reboot → exit MM, rather than racing the whole cluster in one night. On a large migration program we moved VMs in three nightly waves (~50–70 VMs per night) ahead of host work so evacuate did not turn into surprise Storage vMotion during business hours. Preflight also threw “Compatibility check took too long” on a busy vCenter — check vCenter logs, reduce concurrent tasks, retry off-peak. Treat timeout as a signal, not noise. Order I actually runStep What1 Interop matrix: vCenter, ESXi, UCS firmware bundle2 Host profiles → 6.7+3 vCenter upgrade (with extension/monitoring plan)4 Rebuild Lifecycle Manager baselines on new vCenter5 ESXi hosts: vendor bundle, MM, evac, patch, verify6 Storage/cluster validation if VMs use Veritas or RDMThat last row matters if you are also moving SAN paths — see Veritas SAN migration. ESXi maintenance and storage cutover in the same week without coordination is how plexes go TEMPRMSD. Quick referencePreflight / error First moveHost profile version Upgrade profiles to 6.7+ (KB 52932)Extension warning Confirm Aria/vROps compatibility; plan re-registerDRS fully automated Manual/partial for upgrade windowESXi dependency error Vendor bundle; check orphan VIBs vs active HBAsRemote depot URL Copy zip to VMFS firstvCenter DB export fail Stop; support bundle; do not blind retryRelated: Cisco UCS notes · VMware · From VMware to Kubernetes · Veritas SAN migration · Physical hardware

Cisco UCS — Notes From the Floor (Not a Tutorial)

This post sits between two other topics on the blog:vCenter 8 / ESXi 8 preflight — Cisco ESXi images and host patching Physical hardware — cabling, SAN boot, and rack workFull pages: VMware · Physical Hardware This is not a UCS Manager walkthrough. These are the things I keep running into on real UCS builds in VMware shops. What UCS does in a VMware environment UCS is more than managing a blade servers. Most of the important work lives in UCS Manager:Service profiles Pools VLAN groups Firmware bundlesAll of that has to match what ESXi and the SAN expect on the other end of the cable. For initial set up, I spend time on service profile build, Fabric Interconnect (FI) cluster setup, firmware alignment with ESXi, and fixing issues when vMotion or SAN zoning does not work after a change. Service profiles: keep them the same Two ESXi hosts in the same cluster should not be built manually. Slight difference in vNIC order, BIOS settings, or one-off WWPN mappings cause problems later. What I do:Build one template profile for the cluster (management, vMotion, VM traffic, and SAN). Copy it for each host. Only change what must change, like the name or pool assignment. Write the profile name on the build sheet next to switch ports and WWPNs.Saving five minutes with a custom one-off profile usually costs hours later. Fabric Interconnect clustering: match firmware first When you add a second FI to a cluster, the setup checks local vs peer firmware. If they do not match, the join stops. Sometimes the console offers to update the local FI to match the peer. Make sure the peer FI is already stable on the target version. Do not interrupt the FI firmware update with Ctrl+C. The warning is there for a reason.After both FIs are up, UCS Manager is not fully useful until both sides of the cluster are configured: cluster VIP, fabric A/B management IPs, and DNS/domain if you use them. One FI alone is only half the control plane. A few setup mistakes I have learned the hard way:Strong password policy on the FI admin account — a typo during setup can mean starting over. Fabric A vs Fabric B — picking the wrong one is painful to fix later. The cluster IP is not the same as each FI's mgmt0 IP. Write both down before you clean up cables.Firmware bundles must match everything else UCS firmware does not match all on its own. The bundle we pick in UCS Manager must match:Your FI, IOM, and blade or Chassis hardware The ESXi version on the hosts (and the Cisco custom ESXi image if you use it) Any SAN or NIC features you rely on, like SR-IOV or boot policyUpdating one blade with a random BIOS or adapter change may look fine until that one host fails vMotion or behaves differently under load. I update at the bundle level per cluster, the same way I patch ESXi in waves. For ESXi 8 on UCS, I use the Cisco UCS add-on depot profile, not the generic VMware zip. Same lesson as the ESXi upgrade post. Uplinks, optics, and link errors When uplinks flap or performance is odd, I check the optics before I blame ESXi: show interface ethernet <uplink-range> transceiver details show interface ethernet <uplink-range> counters errorsGood Rx/Tx power and zero FCS/CRC errors do not prove the full path is clean end to end. But bad light levels or rising error counters on FI uplinks are worth fixing before you open a VMware support case. Label fiber both ends (FI port → patch panel → core switch port). Write down the full physical path — not just the VLAN name in UCS Manager. VLAN work and a small tool I built Adding VLANs to the right VLAN group in UCS Manager is repetitive and easy to get wrong. I built a small UCS Infra Console (PowerShell + Cisco.UCSManager, optional Docker on Alma/RHEL 9 when time permits will try to add a seperate post on it) that:Reads VLAN and VLAN-group membership from UCS Manager Applies bulk adds from a CSV (VlanName, VlanGroup, VlanId when creating new VLANs) Commits the change in a UCS transaction when you check ApplyIt does not replace UCS Manager. It just saves time on work I do often. SAN handoff: FI is only one step Before I call an ESXi host ready, the build sheet should include:WWPNs from the service profile (both fabrics) FI → MDS/core port mapping (which uplink goes where) Storage team confirmation on both fabrics and all pathsThe most common SAN-boot failure I still see is a wrong WWPN on the ticket compared to what is on the HBA. UCS was configured correctly. The spreadsheet was not. If the host runs Veritas or other cluster software on shared LUNs, coordinate storage work with ESXi maintenance. See Veritas SAN migration. Intersight vs UCSM I Work mostly on UCS Manager but I watch Intersight compatibility for planning, but day-to-day work stays in UCSM, the FI CLI, and PowerShell against the UCS Manager API. Quick referenceProblem Check firstSecond FI will not join the cluster Local vs peer FI firmware / UCS Manager versionOne blade behaves differently Service profile differences, pool assignment, BIOS policyvMotion fails after a UCS change vNIC order, VLAN group, MTU on FI uplinksSAN path missing WWPN on ticket vs profile vs multipath -llNew VLAN needed in production VLAN group membership — typos often show up only at deploy timeRelated: Physical hardware · vCenter 8 / ESXi 8 preflight · VMware · Veritas SAN migration