In N5K-C5596UP, i see all ports in "faulty". Is this something HW?

[Putra]

Unexpectedly, started seeing all ports stutus in "Faulty"

switch# sh interface status

--------------------------------------------------------------------------------
Port Name Status Vlan Duplex Speed Type
--------------------------------------------------------------------------------
Eth1/1 to-l0 faulty 1 full 10G Fabric Exte
Eth1/2 to-10 faulty 1 full 10G Fabric Exte
Eth1/3 to-10 faulty 1 full 10G Fabric Exte
Eth1/4 to-10 faulty 1 full 10G Fabric Exte
Eth1/5 to-10 faulty 1 full 10G Fabric Exte
Eth1/6 to-10 faulty 1 full 10G Fabric Exte

*****output truncated

switch# sh module
Mod Ports Module-Type Model Status
--- ----- ----------------------------------- ---------------------- -----------
1 48 O2 48X10GE/Modular Supervisor N5K-C5596UP-SUP active *
2 16 O2 16 port flexible GEM N55-M16UP ok
3 16 O2 16 port flexible GEM N55-M16UP ok
4 0 O2 GEM with L3 ASIC N55-M160L3-V2 offline

Does reboot fix this issue?

[M02@rt37]

Hello @Putra 

Before considering a reboot, it's advisable to investigate the root cause of the issue. The "Faulty" status on all ports may indicate a hardware or software problem.

#show logging ; Does this output show you something more about this issue ?

The "offline" status of module 4 (O2 GEM with L3 ASIC N55-M160L3-V2) "could be" the root cause of the issue. This module likely handles L3 functions, and if it's offline, it could affect the overall functionality of the switch, leading to the "Faulty" status on all ports.

[Putra]

Thank you for reply. Below is the show-tech output:

Module 4: O2 GEM with L3 ASIC SerialNo : FOC17UUXX

Overall Diagnostic Result for Module 4 : PASS
Diagnostic level at card bootup: complete

Test results: (. = Pass, F = Fail, I = Incomplete,
U = Untested, A = Abort)

1) TestSPROM ---------------------------> .
2) TestLED -----------------------------> .
3) TestTemperatureSensor ---------------> .
4) TestFabricEngine :

Mod Model Power Current Power Current Status
Requested Requested Allocated Allocated
(Watts) (Amps) (Watts) (Amps)
--- ---------------------- ------- ---------- --------- ---------- ----------
1 N5K-C5596UP-SUP 648.00 54.00 648.00 54.00 powered-up
2 N55-M16UP 90.00 7.50 90.00 7.50 powered-up
3 N55-M16UP 90.00 7.50 90.00 7.50 powered-up
4 N55-M160L3-V2 110.40 9.20 110.40 9.20 fail/shutdown

Model 4 logs:

%VPC-2-ASIC_FAILURE_NOTIF: ASIC failure received from nohms in domain 1
%PFMA-5-MOD_DETECT: Module 4 detected (Serial number FOC17UUXX) Model-Type O2 GEM with L3 ASIC Model N55-M160L3-V2

[balaji.bandi]

Since no port working - you can reboot the switch - connect console cable and post complete logs until switch boot completly .

other side  Looks for me Bug :

https://quickview.cloudapps.cisco.com/quickview/bug/CSCur03134

similar troubleshoot suggest raise an TAC case :

https://www.cisco.com/c/en/us/support/docs/switches/nexus-5000-series-switches/116247-problemsolution-product-00.html

[M02@rt37]

@Putra 

Thanks for that output.

Since all ports shows faulty, proceed to a reboot of that unit. Check this reboot in console and trace the boot process.

[balaji.bandi]

what nexus code running on this 5K

how long was the uptime ? did all the ports show faulty even eth 2/X ? and eth3/X ?

is this issue occurred when the device live ? or never worked ? or first time installation ?

as suggested upgrade to latest code 7.1 or above to fix the issue, still issue then RMA is the suggestion only effected cards.

this 5596 is modular - one of module show as offline and try to reseat and check the logs.

[Putra]

Hi BB, Thank you for replying.  The unit is running with version 7.1(4)N1(1). Kernel uptime is 1811 day(s). Yes, all ports shows faulty Eth1/x 2/x and 3/x. No activity performed on this unit, its an unexpected incident. The peer(A-side) live without any issue.