How does VMware assign vmnic labels to Cisco UCS blade servers?

[JDMils]

I have around 10 UCS B200M4 blades in a VMware cluster. They all use the following setup for networking in VMware:

Local dvSwitch:

Management: vmnic4 & vmnic5

Distributed dvSwitch:

Storage: vmnic1 & vmnic3

Data: vmnic0 & vmnic2

In UCS Manager, the NICs above are configured as follows:

MLOM NIC 1 = ESXi_Data_A

MLOM NIC 2 = MGT_B

MLOM NIC 3 = STO_A

VIC NIC 1 = ESXi_Data_B

VIC NIC 2 = MGT_A

VIC NIC 3 = STO_B

I added a new blade to the chassis, B200M4, created a Service Profile from the same template as the existing servers, yet I got the following configuration:

Local dvSwitch:

Management: vmnic1 & vmnic4

Distributed dvSwitch:

Storage: vmnic2 & vmnic5

Data: vmnic0 & vmnic3

In UCS Manager, the NICs above are configured as follows:

MLOM NIC 1 = ESXi_Data_A

MLOM NIC 2 = MGT_B

MLOM NIC 3 = STO_A

VIC NIC 1 = ESXi_Data_B

VIC NIC 2 = MGT_A

VIC NIC 3 = STO_B

Ideally I wanted the new server to use the same "vmnic" labels in the respective switches, as in the other hosts, but for some reason, it seems that the NICs were presented to VMware in a different order and thus given different vmnic labels.

I'd like to understand what I did wrong and how I can avoid this in the future?

[josoneal]

Hello JDMills,

You have done nothing wrong. This has been a known issue where vnics on ESXi host are mismatched from UCSM Service profile. We do have a bug filed for this: https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuv34051
You can try the workarounds, but we have found that those workaround aren't always successful. Reinstalling the OS has a better success rate on resolving this issue. Cisco and VMWare have worked on resolving this issue, but due to the nature of how UCSM presents the vnics and how ESXi determines the vnic naming, we have yet to resolve this issue.
Here is a KB article by VMWare on how VMware ESXi determines the order in which names are assigned to devices: https://kb.vmware.com/s/article/2091560

-Josh

[JDMils]

Thank you Josh, your reply was very helpful.

[Fred_vBrain]

Just for information. I encountered this issue also on a B200M5 with 3.2(3j).

The issue is also depending on the Board Controller Firmware.

In my case I have 2 versions. v11 and v12. The reason for that is that 3.2(3j) is released with v11 but newer UCS blades came with v12.

When you install ESXi on a v11 Board Controller version you get the following order:

vHBA1 -> vmhba0

onboard SATA controller -> vmhba1

vHBA2 -> vmhba2

With a v12 version you get this:

onboard SATA controller -> vmhba0

vHBA1 -> vmhba1

vHBA2 -> vmhab2

Although all PCU bus addresses are the same it will be discovered for whatever reason differently depending on the Onboard Controller firmware version.

Cheers,

Fred

[gman66]

I have the same issue, although we got the proper order we wanted in vSPhere, we achieved this by adding the vNICs to the same vCON, vCON1, in the order we want them to appear.  HOWEVER, the MAC addresses were juggled around in the process so the order being correct is completely irrelevant.  We have the right order we wanted, but the MAC address changes create a real problem! 

[JDMils]

Hey GMan, not sure if you are seeing the same issue as mine. The UCS vNICs always have the same MACs, it's just the ESXi Host vmnics are different from host to host after being deployed from the same Service Profile Template.

For example, this issue hit again recently. We now have NSX-T and the network team configured the Transport Node Profile to automatically assign vmnics2 & 5 to the NSX-T NVDS switch. I setup a new NSX-T-based UCS Service Profile Template based on this requirement, and deployed it to 3 B200M4s and one B200M5. On the M4s, vmnics2 & 5 had the correct VLAN backing as defined in the UCS SPT and thus the hosts worked OK, however on the M5, although the vmnics2 & 5 were automatically assigned to the NSX-T NVDS switch by the NSX-T policy, in the back-end UCS physical server, they were actually Management & Data vNICs and not NSX-T vNICs so that did not work. Same UCS back-end config, just a slightly different hardware revision and the vmnics came out in the wrong order.

MAC addresses should not change on the UCS vNICs, however when viewed in the ESXi OS, the vNIC order is different, maybe this is what you are seeing.

Anyway, I've fix the issue from the ESXi OS, by swapping the vmnics around using this set of instructions: https://kb.vmware.com/s/article/2091560. It allows me to swap the vmnic names around in the ESXI OS so that the intended vmnic name appears connected to the intended UCS vNIC.

[gman66]

Hi Steve - perhaps you could help make sense of what I see regarding CDN.   It is set at the platform default, but when I go to verify what this actually means I don't get a straight forward answer.  See screenshot below.   Although it says the default is enabled, the dependencies tell a different story, and I am not certain what this story means.   Disabled in UCSM but enabled in standalone mode??