Multiple VSMs with different domain IDs cause loop/traffic loss

[mcneilj]

I'm trying to set up two different Nexus 1000v installations in two different clusters in the same vCenter.  I've had one running for a few months:

TestCluster

• Domain ID: 950
• Two VSMs, primary and standby, deployed on separate hosts within the cluster.
• VEMs installed only on hosts in this cluster.
• Management VLAN 1
• Control and Packet share L2 VLAN 950 (unrouted)

This setup worked fine until I tried to add the second installation:

ProdCluster

• Domain ID: 951
• Two VSMs, primary and standby, deployed on separate hosts within the cluter.
• VEMs installed only on hosts in this cluster.
• Management VLAN 1 (Same management VLAN as TestCluster VSMs)
• Control and Packet share L2 VLAN 951 (unrouted)

When the ProdCluster VSMs are brought online, VLAN 1 acts like there is a loop occurring.  There is a large amount of traffic on that VLAN and it appears that there are duplicate packets when sniffing the traffic.  I haven't been able to examine traffic from the Control and Packet VLANs to see what is going on in them.  From what I've read this is a supported configuration.  The only thing I can think that might be causing the problem is that I'm using the same vmware port-group names for my Control, Packet and other networks.  Would this confuse vCenter and make it maybe bridge VLANs together that shouldn't be?  Are there any other configuration items I could check?  I throught maybe there was some duplicate MAC address both were using but that doesn't appear to be happening.

[sachthak]

Hi Jim,

A couple quick questions:

Are your VSM's using portgroups on the vSwitch or the N1k?

Are both clusters in the same vCenter datacenter?

If the above answers are both yes, I can see there being issues since a portgroup with the same name on the vSwitch maps to a common vCenter managed object reference ID. This can be verified by checking the Managed object IDs in the VC MOB (accessible via the browser at https://<VC IP>/mob)

Hope this helps!

Cheers,

Sachin

[mcneilj]

Hi Sachin,

Yes, the VSMs are using port groups on the N1k and both clusters are in the same vCenter datacenter.  I'm not familiar with navigating the MOB.  Do you know where in the tree I'd need to look to view the port group IDs?

I'm going to try building a new ProdCluster VSM with different port group names just to be certain it gets new IDs to see if that resolves the issue.

Thank you for your assistance!

Jim