Nexus back-to-back vPC design with two root ports

[Michiel Vercoutter]

Hi,

I'm running into a scenario where a Nexus switch has two root ports and I'm trying to figure out:

- if this is normal behavior

- whether or not I have a mistake in my configuration

- what would be the impact on traffic flow with two root ports

Design:

Back-to-back vPC design

Running MST

Port-channel 1 on each switch is the peer-link

Port-channel 25 is the port-channel between the two vPC domains

It is not possible to add extra connections between the vPC domains

BE-PLW-COR-001 should be the root

Switch model: C9348GC-FXP

NXOS: 9.3(8)

Config:

BE-PLW-COR-001:

spanning-tree mode mst

spanning-tree mst 0-4094 priority 4096

vpc domain 1
peer-switch
role priority 1
system-priority 50
peer-keepalive destination 1.1.1.2 source 1.1.1.1 vrf VPC-KEEPALIVE
delay restore 150
peer-gateway
layer3 peer-router
auto-recovery
ip arp synchronize

interface port-channel1
switchport mode trunk
spanning-tree port type network
vpc peer-link

interface port-channel25
switchport mode trunk
switchport trunk allowed vlan 21-22,31-32,1007-1009,1107-1108,1509,1609,1709,1826-1829,2021,2209,2727-2728
vpc 25

BE-PLW-COR-002:

spanning-tree mst configuration

spanning-tree mst 0-4094 priority 4096

vpc domain 1
peer-switch
role priority 2
system-priority 50
peer-keepalive destination 1.1.1.1 source 1.1.1.2 vrf VPC-KEEPALIVE
delay restore 150
peer-gateway
layer3 peer-router
auto-recovery
ip arp synchronize

interface port-channel1
switchport mode trunk
spanning-tree port type network
vpc peer-link

interface port-channel25
switchport mode trunk
switchport trunk allowed vlan 21-22,31-32,1007-1009,1107-1108,1509,1609,1709,1826-1829,2021,2209,2727-2728
vpc 25

BE-PLW-COR-003:

spanning-tree mst configuration

spanning-tree mst 0-4094 priority 12288

vpc domain 2
peer-switch
role priority 1
system-priority 50
peer-keepalive destination 1.1.1.2 source 1.1.1.1 vrf VPC-KEEPALIVE
delay restore 150
peer-gateway
layer3 peer-router
auto-recovery
ip arp synchronize

interface port-channel1
switchport mode trunk
spanning-tree port type network
vpc peer-link

interface port-channel25
switchport mode trunk
switchport trunk allowed vlan 21-22,31-32,1007-1009,1107-1108,1509,1609,1709,1826-1829,2021,2209,2727-2728
vpc 25

BE-PLW-COR-004:

spanning-tree mst configuration

spanning-tree mst 0-4094 priority 12288

vpc domain 2
peer-switch
role priority 2
system-priority 50
peer-keepalive destination 1.1.1.1 source 1.1.1.2 vrf VPC-KEEPALIVE
delay restore 150
peer-gateway
layer3 peer-router
auto-recovery
ip arp synchronize

interface port-channel1
switchport mode trunk
spanning-tree port type network
vpc peer-link

interface port-channel25
switchport mode trunk
switchport trunk allowed vlan 21-22,31-32,1007-1009,1107-1108,1509,1609,1709,1826-1829,2021,2209,2727-2728
vpc 25

Spanning-tree output:

BE-PLW-COR-001:

sh spanning-tree mst 0

##### MST0 vlans mapped: 1-1208,1210-1308,1310-2008,2010-2020,2022,2025-2108,2110-4094
Bridge address 0023.04ee.be01 priority 4096 (4096 sysid 0)
Root this switch for the CIST
Regional Root this switch
Operational hello time 2 , forward delay 15, max age 20, txholdcount 6
Configured hello time 2 , forward delay 15, max age 20, max hops 20

Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1 Desg FWD 100 128.4096 (vPC peer-link) Network P2p
Po25 Desg FWD 200 128.4120 (vPC) P2p

BE-PLW-COR-002:

sh spanning-tree mst 0

##### MST0 vlans mapped: 1-1208,1210-1308,1310-2008,2010-2020,2022,2025-2108,2110-4094
Bridge address 0023.04ee.be01 priority 8192 (8192 sysid 0)
Root address 0023.04ee.be01 priority 4096 (4096 sysid 0)
port Po1 path cost 0
Regional Root address 0023.04ee.be01 priority 4096 (4096 sysid 0)
internal cost 100 rem hops 19
Operational hello time 2 , forward delay 15, max age 20, txholdcount 6
Configured hello time 2 , forward delay 15, max age 20, max hops 20

Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1 Root FWD 100 128.4096 (vPC peer-link) Network P2p
Po25 Desg FWD 200 128.4120 (vPC) P2p

BE-PLW-COR-003:

sh spanning-tree mst 0

##### MST0 vlans mapped: 1-1208,1210-1308,1310-2008,2010-2020,2022,2025-2108,2110-4094
Bridge address 0023.04ee.be02 priority 12288 (12288 sysid 0)
Root address 0023.04ee.be01 priority 4096 (4096 sysid 0)
port Po25 path cost 0
Regional Root address 0023.04ee.be01 priority 4096 (4096 sysid 0)
internal cost 200 rem hops 19
Operational hello time 2 , forward delay 15, max age 20, txholdcount 6
Configured hello time 2 , forward delay 15, max age 20, max hops 20

Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1 Desg FWD 100 128.4096 (vPC peer-link) Network P2p
Po25 Root FWD 200 128.4120 (vPC) P2p

BE-PLW-COR-004:

sh spanning-tree mst 0

##### MST0 vlans mapped: 1-1208,1210-1308,1310-2008,2010-2020,2022,2025-2108,2110-4094
Bridge address 0023.04ee.be02 priority 16384 (16384 sysid 0)
Root address 0023.04ee.be01 priority 4096 (4096 sysid 0)
port Po1 path cost 0
Regional Root address 0023.04ee.be01 priority 4096 (4096 sysid 0)
internal cost 300 rem hops 18
Operational hello time 2 , forward delay 15, max age 20, txholdcount 6
Configured hello time 2 , forward delay 15, max age 20, max hops 20

Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1 Root FWD 100 128.4096 (vPC peer-link) Network P2p
Po25 Root FWD 200 128.4120 (vPC) P2p

Any help is greatly appreciated.

[Sergiu.Daniluk · Accepted solution]

Hi @Michiel Vercoutter 

What you see there is expected. Peer-link has some very special attributes when it comes to STP.

The most important one is that the Peer-Link can NEVER be in blocking state. Now since one side is in Designated, the only reasonable option left on the peer side is Root role. 

Take care,

Sergiu

[Michiel Vercoutter]

Alright, makes sense! Thank you Sergiu.

[Kapil Kulkarni]

@Sergiu.Daniluk 

So things would work & there won't be a loop. right ?

[Sergiu.Daniluk]

Correct. vPC peer-link, despite being in FWD state all the time (exception when Bridge Assurance kicks in), it will not create a loop. There is a built-in loop avoidance mechanism. If you are interested in reading more about it I would recommend this document: https://www.cisco.com/c/dam/en/us/td/docs/switches/datacenter/sw/design/vpc_design/vpc_best_practices_design_guide.pdf (section vpc loop avoidance)

Take care,

Sergiu

[Kapil Kulkarni]

@Sergiu.Daniluk 

Thanks a ton for your quick respone..Really appreciate that 

[Michiel Vercoutter]

@Kapil Kulkarni I can confirm this setup works as expected

[Kapil Kulkarni]

@Michiel Vercoutter 

Thanks for confirmation!!!