Connect 4 Nexus3548 via vPC in 2 separated Racks

[Michael Muenz]

Hey guys,

normally we use Catalysts with Stacking to interconnect redundant Switches via different Racks. So there would be Stack1 (2 Switches) in Rack1 and Stack2 (2 Switches) in Rack2. Connectivity between 2 racks is 4x10GB. So I can use Multichassis Etherchannel to get 40GB.

Now we plan to use 2 Nexus 3548 in Rack1 and 2 3548 in Rack2. Since they don't support stacking, I'd create a 2x10GB vPC between switches in same rack and 2x 2x10GB between the rack. 

Q1: Is this possible?

Q2: How do I configure the servers with redundant NICs? LACP and connect to both switches in the same rack?

I included a small picture with the setup.

Thanks!

[Michael Muenz]

No ideas? 

For Q2 I found that LACP is ok to interconnect to two switches like with MEC. 

But does this setup like in the picture makes sense? 

[Reza Sharifi]

Hi,

You can only have 2 switches in a VPC domain and each VPC domain has to have a unique id.

So, looking at your diagram, if the 2 lower switches are you access switches and the 2 upper switches are your core switches than you want one vpc domain for the 2 lower switches and one vpc domain for the 2 upper switches (i.e 100 and 200). You can than connect your servers to both lower switches (access layer) using LACP. I don't think you can connect a server to 2 different VPC domains as shown in your diagram but can't be 100% sure.

HTH

[Michael Muenz]

Hi Reza,

thanks for your answer!

What I want to achieve is, that the switches in Rack1 act as one logical unit and the switches in Rack2 act as one logical unit. So I'd create one vPC Domain in Rack1 (100) and one in Rack2 (200). Then I can connect a server via LACP to both switches in same rack. 

But to be fully redundant between the racks, I need a connection beween logical switch1 and logical switch2. Is this a third domain or does this not work?

Thanks! 

Michael

[Reza Sharifi · Accepted solution]

Hi Michael,

What I want to achieve is, that the switches in Rack1 act as one logical unit and the switches in Rack2 act as one logical unit. So I'd create one vPC Domain in Rack1 (100) and one in Rack2 (200). Then I can connect a server via LACP to both switches in same rack.

What you described here works just fine. What I would add to that is if you want to have an extra layer of redundancy, put each member of vpc domain in a different rack. So, vpc 100 will have switch in say rack-1 and rack-2 and vpc 200 will have a member in rack-1 and rack-2.  This way for whatever reason if something happens to one of the racks (power issue, water leak, etc) you still have the other member in the other rack working. Of course, this requires purchasing some longer cables, because now the servers need to go to different racks and not to just one but it maybe worth looking at it.

But to be fully redundant between the racks, I need a connection beween logical switch1 and logical switch2. Is this a third domain or does this not work?

For connectivity between logical switch1 and 2, there is no need for a new domain.  You just need VPC with Portchannels.

HTH

[Michael Muenz]

HI Reza,

Great! The racks are located in different datacenter rooms with 4x OM4 available, so I can't separate the switches, but the servers (both ESX) do their own HA, so I think I'm fine. Thanks for your help Reza!

Michael 

[Michael Muenz]

Hi Reza,

now I have the switches set up, thanks! 

But how can I check if it really uses the 40GB? On each switch I see a Portchannel with 20GB. Is there a "show vpc XXX" command to see if the Channel is up on all ports.

Thanks!

Michael

[Peter Koltl]

 What you concluded is perfect but let me add one more thing.

With NIC teaming (no LACP) either NIC of a Linux/Windows server can be connected to any switch, even to two different logical switches and they don't require a port-channel on the Cisco side. But If it's vSphere, a port-channel is required so you should connect both links to the same logical switch.

[Reza Sharifi]

Peter,

Can you clarify when you say "With NIC teaming (no LACP) either NIC of a Linux/Windows server can be connected to any switch" do you mean physical servers?

And when you say "But If it's vSphere, a port-channel is required" do you mean virtual servers (VMs)?

If the answer to both questions is yes, can you NIC team on the VMs side as well so, there is no need for Portchannel?

Thanks,

[Michael Muenz]

Peter, 

Which teaming mode do you mean? For Linux there are modes 0-6 (one includes LACP).

Thanks

Michael

[Peter Koltl]

Yes I meant physical servers for Windows/Linux.

And I meant physical vSphere server's vmnic redundancy.

With regard to a VM's vNIC teaming thus not requiring a port-channel...It is not common practice as redundancy should be usually handled by the hypervisor and a single vNIC is enough. But if you use two vNICs with two vSwitches and two vmnics, I guess it would work but a bit clumsy.

[Peter Koltl]

For linux I mean the teaming mode that does not cause the server MAC address flapping between team members, i. e.:

Bond Mode 1 – Active-Backup