Showing Posts From

Fabric interconnect

Cisco UCS — Notes From the Floor (Not a Tutorial)

This post sits between two other topics on the blog:vCenter 8 / ESXi 8 preflight — Cisco ESXi images and host patching Physical hardware — cabling, SAN boot, and rack workFull pages: VMware · Physical Hardware This is not a UCS Manager walkthrough. These are the things I keep running into on real UCS builds in VMware shops. What UCS does in a VMware environment UCS is more than managing a blade servers. Most of the important work lives in UCS Manager:Service profiles Pools VLAN groups Firmware bundlesAll of that has to match what ESXi and the SAN expect on the other end of the cable. For initial set up, I spend time on service profile build, Fabric Interconnect (FI) cluster setup, firmware alignment with ESXi, and fixing issues when vMotion or SAN zoning does not work after a change. Service profiles: keep them the same Two ESXi hosts in the same cluster should not be built manually. Slight difference in vNIC order, BIOS settings, or one-off WWPN mappings cause problems later. What I do:Build one template profile for the cluster (management, vMotion, VM traffic, and SAN). Copy it for each host. Only change what must change, like the name or pool assignment. Write the profile name on the build sheet next to switch ports and WWPNs.Saving five minutes with a custom one-off profile usually costs hours later. Fabric Interconnect clustering: match firmware first When you add a second FI to a cluster, the setup checks local vs peer firmware. If they do not match, the join stops. Sometimes the console offers to update the local FI to match the peer. Make sure the peer FI is already stable on the target version. Do not interrupt the FI firmware update with Ctrl+C. The warning is there for a reason.After both FIs are up, UCS Manager is not fully useful until both sides of the cluster are configured: cluster VIP, fabric A/B management IPs, and DNS/domain if you use them. One FI alone is only half the control plane. A few setup mistakes I have learned the hard way:Strong password policy on the FI admin account — a typo during setup can mean starting over. Fabric A vs Fabric B — picking the wrong one is painful to fix later. The cluster IP is not the same as each FI's mgmt0 IP. Write both down before you clean up cables.Firmware bundles must match everything else UCS firmware does not match all on its own. The bundle we pick in UCS Manager must match:Your FI, IOM, and blade or Chassis hardware The ESXi version on the hosts (and the Cisco custom ESXi image if you use it) Any SAN or NIC features you rely on, like SR-IOV or boot policyUpdating one blade with a random BIOS or adapter change may look fine until that one host fails vMotion or behaves differently under load. I update at the bundle level per cluster, the same way I patch ESXi in waves. For ESXi 8 on UCS, I use the Cisco UCS add-on depot profile, not the generic VMware zip. Same lesson as the ESXi upgrade post. Uplinks, optics, and link errors When uplinks flap or performance is odd, I check the optics before I blame ESXi: show interface ethernet <uplink-range> transceiver details show interface ethernet <uplink-range> counters errorsGood Rx/Tx power and zero FCS/CRC errors do not prove the full path is clean end to end. But bad light levels or rising error counters on FI uplinks are worth fixing before you open a VMware support case. Label fiber both ends (FI port → patch panel → core switch port). Write down the full physical path — not just the VLAN name in UCS Manager. VLAN work and a small tool I built Adding VLANs to the right VLAN group in UCS Manager is repetitive and easy to get wrong. I built a small UCS Infra Console (PowerShell + Cisco.UCSManager, optional Docker on Alma/RHEL 9 when time permits will try to add a seperate post on it) that:Reads VLAN and VLAN-group membership from UCS Manager Applies bulk adds from a CSV (VlanName, VlanGroup, VlanId when creating new VLANs) Commits the change in a UCS transaction when you check ApplyIt does not replace UCS Manager. It just saves time on work I do often. SAN handoff: FI is only one step Before I call an ESXi host ready, the build sheet should include:WWPNs from the service profile (both fabrics) FI → MDS/core port mapping (which uplink goes where) Storage team confirmation on both fabrics and all pathsThe most common SAN-boot failure I still see is a wrong WWPN on the ticket compared to what is on the HBA. UCS was configured correctly. The spreadsheet was not. If the host runs Veritas or other cluster software on shared LUNs, coordinate storage work with ESXi maintenance. See Veritas SAN migration. Intersight vs UCSM I Work mostly on UCS Manager but I watch Intersight compatibility for planning, but day-to-day work stays in UCSM, the FI CLI, and PowerShell against the UCS Manager API. Quick referenceProblem Check firstSecond FI will not join the cluster Local vs peer FI firmware / UCS Manager versionOne blade behaves differently Service profile differences, pool assignment, BIOS policyvMotion fails after a UCS change vNIC order, VLAN group, MTU on FI uplinksSAN path missing WWPN on ticket vs profile vs multipath -llNew VLAN needed in production VLAN group membership — typos often show up only at deploy timeRelated: Physical hardware · vCenter 8 / ESXi 8 preflight · VMware · Veritas SAN migration