Cisco Hyperflex cluster installation failed

[Nice-Infotech]

Hi All, My hyperflex cluster failed with the error "configure Service profile(SP) failed" "Associating Service Profile" during the UCSM configuration in each server but validation was successful. I have 6 nos of hyperflex UCS 240 m5 all-flash nodes and the version of the hyperflex installer is 4.5 2e, UCSM version is 4.2(1n)A and the firmware version of hyperflex nodes is 4.2 (1n)C. I tried other versions of firmware also like 4.1(3m) but it's the same.

Is it a CIMC-related issue or a Server hostname-related issue in DNS?

[riaan.vanniekerk]

Hi there. I doubt it is a DNS issue. The last date of support for Cisco HX Data Platform Software Version 4.5(2x) is December 31 2023 

Is installing HXDP 5.0 an option?

[Nice-Infotech]

Hi, thanks for your suggestion. I install hxdp 5.0 but while validation it is sho

wing error that to check the server port error in Fiber interconnect but there is no issue if I used 4.5.

[Kirk J]

SP association issues are usually related to missing hardware like M.2 boot drive not seated all the way.  Also could be having a 2nd VIC card, that is present but not connected.

Kirk...

[Nice-Infotech]

Hi Kirk, Yes there are two VIC cards in each server one is 1455 and another is 1457. Server ports uplink to FI is connected from VIC 1457 and there is no connectivity from VIC 1455. Is it because of this throwing the error "Server profile association failed" in the installer? Should I Remove this 1455 card or will I connect to FI for redundancy?

Thank you.

 

[Steven Tardy]

Not real sure what step that is in the process and what succeeded before that failure. . . .

Hyperflex installation expects the servers to be discovered and happy (fault-free) within UCSM first.
All Cisco VICs connected to BOTH Fabric Interconnects.
UCS FI ports connected to servers configured as [Server Port].
UCS FI ports connected to upstream configured as [Uplink Port].
All disks being found including internal M.2 boot drive.

Does the installer "see" all of the servers after querying UCSM and allow you to select the servers?

Does the installer create the UCSM Service Profile(s)(SP)?

Are those SP associated to each server?

Do any of those SPs show errors/faults?

Probably need to look at UCSM to see what is / is-not working.

Could be a CIMC issue if CIMC isn't communicating with UCSM properly, but usually not.

Really doubt this is a hostname or DNS issue as the Hyperflex installer can work in absence of DNS by using IPs.

[niceinfotech]

Hi Steven, thanks for your sharing valuable process. Here is my process;

Not real sure what step that is in the process and what succeeded before that failure. . . .

Hyperflex installation expects the servers to be discovered and happy (fault-free) within UCSM first.  ==> Discovered all servers without any error.
All Cisco VICs connected to BOTH Fabric Interconnects. ==> Yes It is connected to both the FIs
UCS FI ports connected to servers configured as [Server Port].==>Yes configured as server port
UCS FI ports connected to upstream configured as [Uplink Port].==>Yes configured as uplink port
All disks being found including internal M.2 boot drive. ==> Yes

Does the installer "see" all of the servers after querying UCSM and allow you to select the servers? ==> Yes

Does the installer create the UCSM Service Profile(s)(SP)?==> Here is the issued, it is failed

Are those SP associated to each server?==>No can not associate it is failed here.

Do any of those SPs show errors/faults?==> Yes: fault is: Service profile rack-unit-6 configuration failed due to insufficient-resources,connection-placement. For all the servers same fault

Probably need to look at UCSM to see what is / is-not working.

Could be a CIMC issue if CIMC isn't communicating with UCSM properly, but usually not. ==> CIMC not configure manually in all the servers, just put the ip pool in the installer. is it because of this but I have another hyperflex cluster is running and up while installing i did not configured cimc not even cabling but it is working.

Really doubt this is a hostname or DNS issue as the Hyperflex installer can work in absence of DNS by using IPs==> Hostname is correctly resolve in the our local DNS.

Please find the screenshot of the error in the hx installer.

[Steven Tardy]

What does the UCSM Service Profile FSM tab show?

Wait. I just saw the UCSM screen shot showing MLOM VIC port 1 and port 3 being connected on the server side but also a second PCIe Cisco VIC above.

Re-cable the second VIC port1 to port3 and then re-ack the server. Then the UCS Service Profile should associate.
If you don't have cables, then physically remove the second PCIe VIC.

ALL Cisco VICs MUST be connected to BOTH Fabric Interconnects. This is a VERY common mistake.

UCSM should show a fault on the server but it does not. I requested an enhancement, but the enhancement got closed because I didn't provide logs as most of TACs labs don't have dual VIC configuration.

If you could provide UCSM logs and one CIMC logs, then I would be forever grateful.