need help with order of vnic

[gman66]

Hi Folks - we have been running VMware on UCS for 5 years now and we are about to move to NFS datastores and retire FC.   I would like to add 2 new vNICs (vmnics in VMWare) to carry NFS traffic on isolated VLANs.  The 2 new vNICs need to be able to carry the NFS storage IO traffic on VLAN x and VLAN y, all other vNICs are configured the same for all other VLANs and in reality could be used interchangeably within VMware since in UCS they all are configured for all VLANs.   So yes, I realize I need to order the vNICs manually so that after boot of the vSphere host the 2 new vmnics (which would be vmnic 4 and 5) would be the ones available for config in the vmmkernel portgroup for NFS I plan to create once these new vmnics are visible within vSphere.  IF the vNICs are not ordered correctly, or all left at any,  they could get re-ordered and negatively impact the stack.   I see the option for manually ordering the vnnic, but I only have 4 virtual slots, and when I manually configure the order whereby vNIC1 is mgmt \ vmnic0, vNIC2 is vmotion vmnic1, vNIC3 and 4 are data plane traffic on vmnic2 and 3, and vNIC 5 and 6 are for NFS on vmnic4 and 5, it doesn't work out because I only have #s 1 through 4, along with "any" to choose from.   there is a section directly above this configuration area that shows the virtual slots, 4 of them,  but the option to add more slots is greyed out.    How can I get these 6 vNICs properly ordered so that VMware continues to work without issues after reboot?  it would seem as though my thinking on how this works is flawed, maybe, but I need some help to sort this out.   

[gman66]

Not a single reply from Cisco here after 70 views???   WOW shocking....    This appears to be a known bug.   I got the vNICs properly ordered in vSPhere by placing them on the same vCON in the order in which I would like them to appear, but the MAC addresses got juggled around making the proper order totally irrelevant.   

What does Cisco have to say about this?   

[Steven Tardy]

This is a known/recurring issue. See my comment on post:
https://community.cisco.com/t5/unified-computing-system-discussions/host-interface-placement-with-two-vics/td-p/4514830

[gman66]

I am aware of that fix in the config file, although VMware took the KB down its still out there on bloggers websites.  That said, Cisco published documentation that states you can prevent all this by manually ordering your vNICs in the service profile, and this works to put the VMNICs in the order you desire, however, the MAC addresses got juggled around and the MAC that was for VMNIC0 is now the MAC address for VMNIC4, a few others changed as well.   The fact that the MAC addresses didn't stay fixed to the vNIC is a Cisco issue, not a VMware issue, this is plain and simple.   

There is another fix, see here, https://kb.vmware.com/s/article/2091560, but will this work to fix the juggling of the MAC addresses?  

Cisco needs to have a way to keep the MAC addresses consistent when the new vNICs are added to the blade, this is pretty ridicuolous, neither HPE nor DELL blades have this issue.    If Cisco doesn't have a fix for this you might as well say goodbye to any further sales or expansion of our chassis 

[Steven Tardy]

Did you configure "CDN" in your UCS Service Profile?
I've seen circumstances where re-installing ESXi is required (again a VMware issue) to get the correct order.

https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/ucs-manager/GUI-User-Guides/Server-Mgmt/4-1/b_Cisco_UCS_Manager_Server_Mgmt_Guide_4_1/4-1-trial_chapter_01100.html#concept_C51EF19CCBDA4AF4B971C0C1373973EB

The reason HPE and Dell blades don't have this issue is because dynamically adding NICs isn't a thing on NICs only on Cisco VICs.
If you were to physically add additional NICs to any server then this "vmnic juggling" can happen.

[gman66]

CDN... I think you just found something...  just now I tried the VMware KB  https://kb.vmware.com/s/article/2091560 to re-assign things but it ended up juggling around the MAC addresses and didn't work as expected.   I will look into this CDN and see what comes of it.   Bottom line, we should have stability in our adapter MAC addresses, they should not change assignments between VMNICs in VMware, even if adding more vNICs, the MAC addresses need to stay put,  this is how communications is being lost.  

[gman66]

There has to be 100 plus customers out there that have run into this issue, I have toyed with a few configurations regarding placement, but the MAC addresses shown in VMWare vmnics still don't align with the vNIC in UCS.   We have actually had this issue from day 1 and never did anything about it because all vNICs were configured to same, so it didn't matter.   We now have to get the placement correct.  

Like I said, there has to be 100s of customers that use the VIC 1340 and 1440 running VMWare that have had to properly order the vNICs so they get enumerated correctly and match up MAC for MAC VNIC to VMNIC.  TAC should also have this info available for customers.   The VMWare OS and Cisco VIC are constants across customer sites, and in the Cisco LABs.  SO therefore it shouldn't be a stretch to get a resolution for this issue.