Showing Posts From

On premises

Cisco UCS — Notes From the Floor (Not a Tutorial)

This post sits between two other topics on the blog:vCenter 8 / ESXi 8 preflight — Cisco ESXi images and host patching Physical hardware — cabling, SAN boot, and rack workFull pages: VMware · Physical Hardware This is not a UCS Manager walkthrough. These are the things I keep running into on real UCS builds in VMware shops. What UCS does in a VMware environment UCS is more than managing a blade servers. Most of the important work lives in UCS Manager:Service profiles Pools VLAN groups Firmware bundlesAll of that has to match what ESXi and the SAN expect on the other end of the cable. For initial set up, I spend time on service profile build, Fabric Interconnect (FI) cluster setup, firmware alignment with ESXi, and fixing issues when vMotion or SAN zoning does not work after a change. Service profiles: keep them the same Two ESXi hosts in the same cluster should not be built manually. Slight difference in vNIC order, BIOS settings, or one-off WWPN mappings cause problems later. What I do:Build one template profile for the cluster (management, vMotion, VM traffic, and SAN). Copy it for each host. Only change what must change, like the name or pool assignment. Write the profile name on the build sheet next to switch ports and WWPNs.Saving five minutes with a custom one-off profile usually costs hours later. Fabric Interconnect clustering: match firmware first When you add a second FI to a cluster, the setup checks local vs peer firmware. If they do not match, the join stops. Sometimes the console offers to update the local FI to match the peer. Make sure the peer FI is already stable on the target version. Do not interrupt the FI firmware update with Ctrl+C. The warning is there for a reason.After both FIs are up, UCS Manager is not fully useful until both sides of the cluster are configured: cluster VIP, fabric A/B management IPs, and DNS/domain if you use them. One FI alone is only half the control plane. A few setup mistakes I have learned the hard way:Strong password policy on the FI admin account — a typo during setup can mean starting over. Fabric A vs Fabric B — picking the wrong one is painful to fix later. The cluster IP is not the same as each FI's mgmt0 IP. Write both down before you clean up cables.Firmware bundles must match everything else UCS firmware does not match all on its own. The bundle we pick in UCS Manager must match:Your FI, IOM, and blade or Chassis hardware The ESXi version on the hosts (and the Cisco custom ESXi image if you use it) Any SAN or NIC features you rely on, like SR-IOV or boot policyUpdating one blade with a random BIOS or adapter change may look fine until that one host fails vMotion or behaves differently under load. I update at the bundle level per cluster, the same way I patch ESXi in waves. For ESXi 8 on UCS, I use the Cisco UCS add-on depot profile, not the generic VMware zip. Same lesson as the ESXi upgrade post. Uplinks, optics, and link errors When uplinks flap or performance is odd, I check the optics before I blame ESXi: show interface ethernet <uplink-range> transceiver details show interface ethernet <uplink-range> counters errorsGood Rx/Tx power and zero FCS/CRC errors do not prove the full path is clean end to end. But bad light levels or rising error counters on FI uplinks are worth fixing before you open a VMware support case. Label fiber both ends (FI port → patch panel → core switch port). Write down the full physical path — not just the VLAN name in UCS Manager. VLAN work and a small tool I built Adding VLANs to the right VLAN group in UCS Manager is repetitive and easy to get wrong. I built a small UCS Infra Console (PowerShell + Cisco.UCSManager, optional Docker on Alma/RHEL 9 when time permits will try to add a seperate post on it) that:Reads VLAN and VLAN-group membership from UCS Manager Applies bulk adds from a CSV (VlanName, VlanGroup, VlanId when creating new VLANs) Commits the change in a UCS transaction when you check ApplyIt does not replace UCS Manager. It just saves time on work I do often. SAN handoff: FI is only one step Before I call an ESXi host ready, the build sheet should include:WWPNs from the service profile (both fabrics) FI → MDS/core port mapping (which uplink goes where) Storage team confirmation on both fabrics and all pathsThe most common SAN-boot failure I still see is a wrong WWPN on the ticket compared to what is on the HBA. UCS was configured correctly. The spreadsheet was not. If the host runs Veritas or other cluster software on shared LUNs, coordinate storage work with ESXi maintenance. See Veritas SAN migration. Intersight vs UCSM I Work mostly on UCS Manager but I watch Intersight compatibility for planning, but day-to-day work stays in UCSM, the FI CLI, and PowerShell against the UCS Manager API. Quick referenceProblem Check firstSecond FI will not join the cluster Local vs peer FI firmware / UCS Manager versionOne blade behaves differently Service profile differences, pool assignment, BIOS policyvMotion fails after a UCS change vNIC order, VLAN group, MTU on FI uplinksSAN path missing WWPN on ticket vs profile vs multipath -llNew VLAN needed in production VLAN group membership — typos often show up only at deploy timeRelated: Physical hardware · vCenter 8 / ESXi 8 preflight · VMware · Veritas SAN migration

Veritas SAN Migration — Mirror First, Remove Second

Sub-topic under Physical Hardware and Veritas clustering. Cluster failover order is one skill. Moving the disks underneath without deleting the only live plex is another — and the second one is where I’ve seen people panic. This is the cutover procedure I run when Veritas Volume Manager (VxVM) volumes on a legacy array need to land on new flash storage — plus what broke when the “remove old disk” step ran too early after a VMware migration. Page: Veritas Clustering. The rule I repeat on every bridge call Mirror onto the new array first. Wait until sync is clean. Only then remove the old side. If you remove the wrong plex while the volume is running on a single path, you don’t get a graceful migration — you get an outage and a very long night. Happy-path cutover (both cluster nodes) Assuming storage has already zoned new LUNs to both nodes and the cluster is healthy:Scan — OS rescan on both nodes, then vxdisk scandisks on each. Initialize new disks — vxdisksetup -i <new_disk> format=cdsdisk Add to disk group — vxdg -g <DG_Name> adddisk <new_disk_name>=<device> Mirror each volume — vxassist -g <DG_Name> mirror <Volume_Name> <new_disk_name>Use -b for background sync if your change window allows it. Watch sync — vxtask list until state is CLEAN; confirm with vxprint -htg <DG_Name>. Remove old mirror — vxassist -g <DG_Name> remove mirror <Volume_Name> <old_disk_name> Drop old disk from DG — vxdg -g <DG_Name> rmdisk <old_disk_name> Reclaim / unmap — storage team unmaps old LUNs only after VxVM layout is clean.Before step 6 I always run vxprint -g <DG_Name> -ht and sanity-check which plex is ACTIVE and RW. Paper diagram beats muscle memory when you’re tired. When the OS sees disks but Veritas doesn’t After a rescan, lsblk may show devices while vxdisk list does not. Typical recovery: # OS-level rescan (repeat per HBA host if needed) echo "- - -" > /sys/class/scsi_host/host0/scan# Veritas rescan vxdisk scandisksIf disks show online invalid or you suspect a clone/duplicate identity after storage work, check for UDID mismatch: vxdisk -v list | grep udid_mismatch vxdisk updateudid <device_name>Skipping that step and forcing adddisk anyway is how you get volumes that look fine until the next reboot. The mistake that almost cost us: TEMPRMSD During a VMware-side migration, one plex went TEMPRMSD (temporarily removed). Veritas had lost access to the underlying disk path, marked that side out of sync, and the volume was effectively running on one plex in WO (write-only) mode. The trap: the plex we wanted to remove was the only active data path. Removing it would have dropped the last good copy. Wrong: “Old array LUN is unmapped — remove that disk from the DG now.”Right: Re-attach and resync the missing plex first, then remove the old side. Recovery pattern: # Re-attach the TEMPRMSD plex and start resync from the good side vxplex -g <DG_Name> att <Volume_Name> <Volume_Name>-02# Monitor until both plexes are ACTIVE / RW vxtask list vxprint -g <DG_Name> -ht <Volume_Name>Only when both sides are healthy: vxplex -g <DG_Name> -o rm dis <Volume_Name>-01 vxdg -g <DG_Name> rmdisk <old_disk_name> vxprint -g <DG_Name> -htThe -o rm on vxplex dis removes the plex structure after dissociating it — don’t run that until you’re sure which side holds production data. How this ties to VMware work Storage migration and vMotion / datastore moves often happen in the same program. VxVM doesn’t know your project plan — it only knows whether both paths to a mirror are still valid. My order when both teams are involved:Confirm multipath clean on both cluster nodes before any VM storage move. Mirror to new array and finish sync before decommissioning old LUNs. Treat any TEMPRMSD or WO plex as a stop sign, not a cleanup shortcut. Failover test after storage cutover, not as a substitute for verifying mirrors.If the cluster mount resource still points at an old device path after re-zoning, fix that before blaming Veritas agents — same theme as Veritas clustering. Quick referenceSymptom First lookVolume on one plex only vxprint -htg <DG_Name> — check ACTIVE vs TEMPRMSDSync stuck or slow vxtask list, backend array performanceDisk visible in OS, not in VxVM Rescan + vxdisk scandisks + UDID checkAbout to rmdisk Confirm no volume layout still references that diskRelated: Veritas clustering · vCenter 8 / ESXi 8 preflight · Cisco UCS notes · Physical hardware · VMware