Troubleshooting and Clearing UCS Faults

Sandeep Singh · ‎10-08-2012

Introduction

This document shows a few common UCS faults and method to clear them using CLI. Since the faults can not be manually deleted through the UCS Manager GUI or CLI, this document shows CLI steps to clear these faults. Each fault represents a failure in the Cisco UCS instance or an alarm threshold that has been raised. During the lifecycle of a fault, it can change from one state or severity to another. Fault includes information about the operational state of the affected object at the time the fault was raised.

Fault Severity

Following table explains fault severity in UCS

Severity	Description
Critical	A service-affecting condition that requires immediate corrective action. For example, this severity could indicate that the managed object is out of service and its capability must be restored
Warning	A potential or impending service-affecting fault that currently has no significant effects in the system. Action should be taken to further diagnose
Major	A service-affecting condition that requires urgent corrective action
Minor	A non-service-affecting fault condition that requires corrective action to prevent a more serious fault from occurring
Condition	An informational message about a condition, possibly independently insignificant
Info	A basic notification or informational message
Cleared	A notification that the condition that caused the fault has beenresolved, and the fault has been cleared

UCS Faults

If port channels and pools are configured in UCS, but not used, they will show up as either a Major or a Minor fault.

UCS1-FA-A# show fault
Severity  Code     Last Transition Time     ID       Description

--------- -------- ------------------------ -------- -----------
Major     F0282    2012-02-08T13:59:23.094   1576322 lan port-channel 2 on fabric interconnect B oper state: failed, reason: No operational members
Major     F0282    2012-02-08T13:58:11.915   1576231 lan port-channel 2 on fabric interconnect A oper state: failed, reason: No operational members
Minor     F0465    2012-04-12T16:52:38.892    193850 IP pool iscsi-initiator-pool is empty
Minor     F0476    2012-05-09T20:21:15.985   1668798 FC pool node-wwn-assignment node-default is empty

Steps required to clear the fault:

Fault #1 - Port Channel with No Operational Members

1. Log onto the primary fabric interconnect through UCS Manager CLI with administrative privileges.
2. Disable the port channel/s with "No operational members".

Fault #2 - IP Pool iscsi-initiator-pool is empty

1. As a UCS user with administrative privileges, create a bogus block of IP address in the iscsi-initiator-pool, such as: 1.1.1.1 to 1.1.1.1.

Fault #3 - FC pool node-wwn-assignment node-default is empty

1. As a UCS user with administrative privileges, create a bogus WWN entry in the node-default pool, such as: 50:00:25:B5:11:11:11:11.

Note: The changes mentioned above are disruptive, please take adequate precautions.

Clear "Port Channel with No Operational Member Fault" on Fabric Interconnect B

UCS1-FA-A# scope eth-uplink

UCS1-FA-A /eth-uplink # scope fabric b

UCS1-FA-A /eth-uplink/fabric # show port-channel

Port Channel:

Port Channel Id Name Admin State Oper State State Reason

--------------- ------------ ----------- ---------------- ------------

1 port_ch_1 Enabled Up

2 port_ch_2 Enabled Failed No operational members

UCS1-FA-A /eth-uplink/fabric # enter port-channel 2

UCS1-FA-A /eth-uplink/fabric/port-channel # disable

UCS1-FA-A /eth-uplink/fabric/port-channel* # commit-buffer

UCS1-FA-A /eth-uplink/fabric/port-channel # exit

UCS1-FA-A /eth-uplink/fabric # show port-channel

Port Channel:

Port Channel Id Name Admin State Oper State State Reason

--------------- ------------ ----------- ---------------- ------------

1 port_ch_1 Enabled Up

2 port_ch_2 Disabled Failed No operational members

UCS1-FA-A /eth-uplink/fabric # exit

UCS1-FA-A /eth-uplink # exit

UCS1-FA-A# show fault

Severity Code Last Transition Time ID Description

--------- -------- ------------------------ -------- -----------

Minor F0476 2012-05-09T20:21:15.985 1668798 FC pool node-wwn-assignment node-default is empty

Major F0282 2012-02-08T13:58:11.915 1576231 lan port-channel 2 on fabric interconnect A oper state: failed, reason: No operational members

Minor F0465 2012-04-12T16:52:38.892 193850 IP pool iscsi-initiator-pool is empty

Notice that the port channel fault on Fabric B is now not there in show fault command output.

Clear "Port Channel with No Operational Member Fault" on Fabric Interconnect A

UCS1-FA-A# scope eth-uplink
UCS1-FA-A /eth-uplink # scope fabric a
UCS1-FA-A /eth-uplink/fabric # enter port-channel 2
UCS1-FA-A /eth-uplink/fabric/port-channel # disable
UCS1-FA-A /eth-uplink/fabric/port-channel* # commit-buffer
UCS1-FA-A /eth-uplink/fabric/port-channel # exit
UCS1-FA-A /eth-uplink/fabric # exit
UCS1-FA-A /eth-uplink # exit
UCS1-FA-A# show fault

Severity  Code     Last Transition Time     ID       Description

--------- -------- ------------------------ -------- -----------

Cleared  F0282    2012-05-11T15:54:27.840   1576231 lan port-channel 2 on fabric interconnect A oper state: failed, reason: No operational members

Minor     F0465    2012-04-12T16:52:38.892    193850 IP pool iscsi-initiator-pool is empty
Minor     F0476    2012-05-09T20:21:15.985   1668798 FC pool node-wwn-assignment node-default is empty

Notice that the port channel fault on Fabric A shows "Cleared".

Clear "IP Pool iscsi-initiator-pool empty" Fault

UCS1-FA-A# scope org
UCS1-FA-A /org # scope ip-pool iscsi-initiator-pool
UCS1-FA-A /org/ip-pool # show
IP Pool:
    Name                 Size       Assigned

    -------------------- ---------- --------

    iscsi-initiator-pool          0          0

UCS1-FA-A /org/ip-pool # create block 1.1.1.1 1.1.1.1 1.1.1.1 255.255.255.0
UCS1-FA-A /org/ip-pool/block* # commit-buffer
UCS1-FA-A /org/ip-pool/block # exit
UCS1-FA-A /org/ip-pool # show

IP Pool:

    Name                 Size       Assigned

    -------------------- ---------- --------

    iscsi-initiator-pool          1        0

UCS1-FA-A /org/ip-pool # show block

Block of IP Addresses:

    From            To              Default Gateway Subnet Mask
    --------------- --------------- --------------- -----------

    1.1.1.1         1.1.1.1         1.1.1.1         255.255.255.0

UCS1-FA-A /org/ip-pool # exit
UCS1-FA-A /org # exit
UCS1-FA-A# show fault

Severity  Code     Last Transition Time     ID       Description
--------- -------- ------------------------ -------- -----------
Cleared  F0465    2012-05-11T17:37:52.178    193850 IP pool iscsi-initiator-pool is empty

Minor     F0476    2012-05-09T20:21:15.985   1668798 FC pool node-wwn-assignment node-default is empty

Notice that the IP pool iscsi-initiator-pool empty fault shows "Cleared".

Clear "FC Pool node-default empty" Fault

UCS1-FA-A /org # show wwn-pool

WWN Pool:

    Name                 Purpose             Size       Assigned
    -------------------- ------------------- ---------- --------
    node-default         Node Wwn Assignment          0        0

UCS1-FA-A /org # scope wwn-pool node-default
UCS1-FA-A /org/wwn-pool # create initiator 50:00:25:B5:11:11:11:11
UCS1-FA-A /org/wwn-pool/initiator* # commit-buffer
UCS1-FA-A /org/wwn-pool/initiator # exit
UCS1-FA-A /org/wwn-pool # exit
UCS1-FA-A /org # exit
UCS1-FA-A# show fault

Severity  Code     Last Transition Time     ID       Description

--------- -------- ------------------------ -------- -----------
Cleared  F0476    2012-05-11T17:46:07.400   1668798 FC pool node-wwn-assignment node-default is empty
Cleared   F0465    2012-05-11T17:37:52.178    193850 IP pool iscsi-initiator-pool is empty

Notice that the FC pool node-default empty fault shows "Cleared".