- Cisco Community
- Technology and Support
- Networking
- Switching
- NX9000 Port channel <> Debian Server Bond Issue
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
NX9000 Port channel <> Debian Server Bond Issue
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-03-2023 08:56 AM
Hey Everyone,
I have an issue as mentioned in the title with an NX9000 port channel config to a Debian Server Bond config. 2 out of the 3
interfaces which are configured in the port channel on the NX (lets call is NX1) are suspended(no LACP PDUs seen on interfaces).
To give an idea on the setup, I have 2 NX9000's (lets call them NX1 and NX2) connected via VPC. The corresponding port channel on NX 2 to the Debian server is up (3/3 ports up). The port channel on NX 1 to the Debian server is up but with (1/3 ports up). The VPC between the NX's are configured correctly:
NX1# sh vpc 4
vPC status
----------------------------------------------------------------------------
Id Port Status Consistency Reason Active vlans
-- ------------ ------ ----------- ------ ---------------
4 Po4 up success success 3304-3305,3463
-3465,3469-347
0,3474,3542-35
43
***************************************************************
NX2# sh vpc 4
vPC status
----------------------------------------------------------------------------
Id Port Status Consistency Reason Active vlans
-- ------------ ------ ----------- ------ ---------------
4 Po4 up success success 3304-3305,3463
-3465,3469-347
0,3474,3542-35
43
On the NX1 to the debian server, the following is seen for the interfaces in the port channel:
NX1# sh run int eth1/3 (Suspended Interface)
!Command: show running-config interface Ethernet1/3
!Running configuration last done at: Mon Jul 3 12:39:03 2023
!Time: Mon Jul 3 14:36:12 2023
version 7.0(3)I7(10) Bios:version 05.38
interface Ethernet1/3
description CDN temp 2
switchport
switchport mode trunk
switchport trunk allowed vlan 3304-3305,3463-3465,3469-3470,3474,3542-3543
channel-group 4 mode active
no shutdown
NX1# sh run int eth2/20 (Suspended Interface)
!Command: show running-config interface Ethernet2/20
!Running configuration last done at: Mon Jul 3 12:39:03 2023
!Time: Mon Jul 3 14:33:46 2023
version 7.0(3)I7(10) Bios:version 05.38
interface Ethernet2/20
description CDN temp 1
switchport
switchport mode trunk
switchport trunk allowed vlan 3304-3305,3463-3465,3469-3470,3474,3542-3543
channel-group 4 mode active
no shutdown
NX1# sh run int eth1/36 (UP interface)
!Command: show running-config interface Ethernet1/36
!Running configuration last done at: Mon Jul 3 12:39:03 2023
!Time: Mon Jul 3 14:34:26 2023
version 7.0(3)I7(10) Bios:version 05.38
interface Ethernet1/36
description CDN Te3 temp
switchport
switchport mode trunk
switchport trunk allowed vlan 3304-3305,3463-3465,3469-3470,3474,3542-3543
channel-group 4 mode active
no shutdown
NX1# sh int status | i sus
Eth1/3 CDN temp 2 suspended trunk auto auto 10Gbase-SR
Eth2/20 CDN temp 1 suspended trunk auto auto 10Gbase-SR
On the lacp neighbor info we see the following (I do notice that the neighbor interfaces with the issues are in SP mode):
ELSO-TV01# sh lacp neighbor interface port-channel 4
Flags: S - Device is sending Slow LACPDUs F - Device is sending Fast LACPDUs
A - Device is in Active mode P - Device is in Passive mode
port-channel4 neighbors
Partner's information
Partner Partner Partner
Port System ID Port Number Age Flags
Eth1/36 65535,ac-1f-6b-9e-1d-b00x3 203073 FA
LACP Partner Partner Partner
Port Priority Oper Key Port State
255 0xf 0x3f
Partner's information
Partner Partner Partner
Port System ID Port Number Age Flags
Eth1/3 0,0-0-0-0-0-0 0x0 203073 SP
LACP Partner Partner Partner
Port Priority Oper Key Port State
0 0x0 0x0
Partner's information
Partner Partner Partner
Port System ID Port Number Age Flags
Eth2/20 0,0-0-0-0-0-0 0x0 203073 SP
LACP Partner Partner Partner
Port Priority Oper Key Port State
0 0x0 0x0
__________________________________________________
This is the internal info for the port-channel with the issue (note you will see more interfaces in the port channel in this result as I tried adding new interfaces in troubleshooting):
To give context in what occurred, the port channel 4 was up and working before but there was an unexpected power outage which took the debian server offline for a short time. Upon booting up, we had this issue only with these interfaces connecting to the server. See below on what we tried:
- Shut/no shut port channel 4
- Shut/no shut suspended interfaces
- Removed suspended interfaces from port-channel (which came up) and re-add members in port-channel (went into suspended state shortly after).
- Added new interfaces to the port channel and connected the ports from the server side to these interfaces but went into suspended state shortly after.
- Changed patch cords and SFPs on both ends.
- Repeated steps 1-3 in varying orders of sequence.
The issue seems to follow the ports on the server as any interface we try on the switch side goes into a suspended state. The 1 interface which is up remains up when we move the interfaces on the switch side and reconfigure with port-channel 4 active mode. We engaged the provider for the server but they said that there is no issue on the server side. They even removed the interfaces from the bond on the server side and reconfigured but no luck. The issue to us really looks like it might be the bond config on the server but I would really like to confirm that the switch side is OK. I would also like to get some help with the server side checks:
This is what the provider was able to send (below 4 of the 6 interfaces are up and the two which are down are enp131s0f1 and enp2s0f0):
DebianServer# ethtool enp2s0f0
Settings for enp2s0f0:
Supported ports: [ FIBRE ]
Supported link modes: 1000baseX/Full
10000baseSR/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 1000baseX/Full
10000baseSR/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Auto-negotiation: off
Port: FIBRE
PHYAD: 0
Transceiver: internal
Supports Wake-on: g
Wake-on: g
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes
DebianServer# ethtool enp131s0f1
Settings for enp131s0f1:
Supported ports: [ FIBRE ]
Supported link modes: 1000baseT/Full
10000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 1000baseT/Full
10000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Auto-negotiation: on
Port: FIBRE
PHYAD: 0
Transceiver: internal
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes
DebianServer# ethtool enp131s0f0
Settings for enp131s0f0:
Supported ports: [ FIBRE ]
Supported link modes: 1000baseT/Full
10000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 1000baseT/Full
10000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Auto-negotiation: on
Port: FIBRE
PHYAD: 0
Transceiver: internal
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes
DebianServer# ethtool ens2f0
Settings for ens2f0:
Supported ports: [ FIBRE ]
Supported link modes: 1000baseT/Full
10000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 1000baseT/Full
10000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Auto-negotiation: on
Port: FIBRE
PHYAD: 0
Transceiver: internal
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes
DebianServer# ethtool enp2s0f1
Settings for enp2s0f1:
Supported ports: [ FIBRE ]
Supported link modes: 1000baseX/Full
10000baseSR/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 1000baseX/Full
10000baseSR/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Auto-negotiation: off
Port: FIBRE
PHYAD: 0
Transceiver: internal
Supports Wake-on: g
Wake-on: g
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes
DebianServer# ethtool ens2f1
Settings for ens2f1:
Supported ports: [ FIBRE ]
Supported link modes: 1000baseT/Full
10000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 1000baseT/Full
10000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Auto-negotiation: on
Port: FIBRE
PHYAD: 0
Transceiver: internal
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes
_____________________________________________
This is the bond file output on server (enp131s0f1 and enp2s0f0 are missing):
Ethernet Channel Bonding Driver: v5.10.0-20-amd64
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 200
Down Delay (ms): 200
Peer Notification Delay (ms): 0
802.3ad info
LACP rate: fast
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: ac:1f:6b:9e:1d:b0
Active Aggregator Info:
Aggregator ID: 1
Number of ports: 4
Actor Key: 15
Partner Key: 32772
Partner Mac Address: 00:23:04:ee:be:0a
Slave Interface: ens2f0
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: ac:1f:6b:9e:1d:b0
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
system priority: 65535
system mac address: ac:1f:6b:9e:1d:b0
port key: 15
port priority: 255
port number: 1
port state: 63
details partner lacp pdu:
system priority: 32667
system mac address: 00:23:04:ee:be:0a
oper key: 32772
port priority: 32768
port number: 589
port state: 61
Slave Interface: ens2f1
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: ac:1f:6b:9e:1d:b1
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
system priority: 65535
system mac address: ac:1f:6b:9e:1d:b0
port key: 15
port priority: 255
port number: 2
port state: 63
details partner lacp pdu:
system priority: 32667
system mac address: 00:23:04:ee:be:0a
oper key: 32772
port priority: 32768
port number: 585
port state: 61
Slave Interface: enp131s0f0
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 1
Permanent HW addr: ac:1f:6b:9e:23:b6
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
system priority: 65535
system mac address: ac:1f:6b:9e:1d:b0
port key: 15
port priority: 255
port number: 3
port state: 63
details partner lacp pdu:
system priority: 32667
system mac address: 00:23:04:ee:be:0a
oper key: 32772
port priority: 32768
port number: 16781
port state: 61
Slave Interface: enp2s0f1
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: ac:1f:6b:c5:7f:ed
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
system priority: 65535
system mac address: ac:1f:6b:9e:1d:b0
port key: 15
port priority: 255
port number: 6
port state: 63
details partner lacp pdu:
system priority: 32667
system mac address: 00:23:04:ee:be:0a
oper key: 32772
port priority: 32768
port number: 597
port state: 61
Any help will be appreciated!
NOTE: If anything isn't clear, please let me know and I will try to clarify. Also if I posted in the wrong board, please let me know!
- Labels:
-
Other Networking
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide
