UCS S3260 M5 - Dual Pass Through Controller based on Broadcom 3316 ROC

[Ozy]

Hello. 

I have UCS S3260 M5 server and my firmware version is 4.2(3e)

I've installed Truenas Scale using (legacy boot) and everything was looking perfect until I reboot the server.

When I reboot the server, at boot sequence I'm getting HBA reset and my zfs pool suspending because of the drive absence.

The story begins here. 

Dmesg output: 

[Sun Jul  9 09:56:01 2023] bond0: (slave eno6): link status definitely up, 10000 Mbps full duplex
[Sun Jul  9 09:56:16 2023] sd 6:0:0:0: device_block, handle(0x0012)
[Sun Jul  9 09:56:16 2023] sd 6:0:1:0: device_block, handle(0x0013)
[Sun Jul  9 09:56:16 2023] sd 6:0:2:0: device_block, handle(0x0014)
[Sun Jul  9 09:56:16 2023] sd 6:0:3:0: device_block, handle(0x0015)
[Sun Jul  9 09:56:16 2023] sd 6:0:4:0: device_block, handle(0x0016)
[Sun Jul  9 09:56:16 2023] sd 6:0:5:0: device_block, handle(0x0017)
[Sun Jul  9 09:56:16 2023] sd 6:0:6:0: device_block, handle(0x0018)
[Sun Jul  9 09:56:16 2023] sd 6:0:7:0: device_block, handle(0x0019)
[Sun Jul  9 09:56:16 2023] sd 6:0:8:0: device_block, handle(0x001a)
[Sun Jul  9 09:56:16 2023] sd 6:0:9:0: device_block, handle(0x001b)
[Sun Jul  9 09:56:16 2023] sd 6:0:10:0: device_block, handle(0x001c)
[Sun Jul  9 09:56:16 2023] sd 6:0:11:0: device_block, handle(0x001d)
[Sun Jul  9 09:56:16 2023] sd 6:0:12:0: device_block, handle(0x001e)
[Sun Jul  9 09:56:16 2023] sd 6:0:13:0: device_block, handle(0x001f)
[Sun Jul  9 09:56:16 2023] sd 6:0:14:0: device_block, handle(0x0020)
[Sun Jul  9 09:56:16 2023] sd 6:0:15:0: device_block, handle(0x0021)
[Sun Jul  9 09:56:16 2023] sd 6:0:16:0: device_block, handle(0x0022)
[Sun Jul  9 09:56:16 2023] sd 6:0:17:0: device_block, handle(0x0023)
[Sun Jul  9 09:56:16 2023] sd 6:0:18:0: device_block, handle(0x0024)
[Sun Jul  9 09:56:16 2023] sd 6:0:19:0: device_block, handle(0x0025)
[Sun Jul  9 09:56:16 2023] sd 6:0:20:0: device_block, handle(0x0026)
[Sun Jul  9 09:56:16 2023] sd 6:0:21:0: device_block, handle(0x0027)
[Sun Jul  9 09:56:16 2023] sd 6:0:22:0: device_block, handle(0x0028)
[Sun Jul  9 09:56:16 2023] sd 6:0:23:0: device_block, handle(0x0029)
[Sun Jul  9 09:56:16 2023] sd 6:0:24:0: device_block, handle(0x002a)
[Sun Jul  9 09:56:16 2023] sd 6:0:25:0: device_block, handle(0x002b)
[Sun Jul  9 09:56:16 2023] sd 6:0:26:0: device_block, handle(0x002c)
[Sun Jul  9 09:56:16 2023] sd 6:0:27:0: device_block, handle(0x002d)
[Sun Jul  9 09:56:16 2023] sd 6:0:28:0: device_block, handle(0x002e)
[Sun Jul  9 09:56:16 2023] sd 6:0:29:0: device_block, handle(0x002f)
[Sun Jul  9 09:56:16 2023] sd 6:0:30:0: device_block, handle(0x0030)
[Sun Jul  9 09:56:16 2023] sd 6:0:31:0: device_block, handle(0x0031)
[Sun Jul  9 09:56:16 2023] sd 6:0:32:0: device_block, handle(0x0032)
[Sun Jul  9 09:56:16 2023] sd 6:0:33:0: device_block, handle(0x0033)
[Sun Jul  9 09:56:16 2023] ses 6:0:34:0: _scsih_block_io_device skip device_block for SES handle(0x0034)
[Sun Jul  9 09:56:16 2023] ses 6:0:34:0: _scsih_block_io_device skip device_block for SES handle(0x0034)
[Sun Jul  9 09:56:16 2023] ses 6:0:34:0: _scsih_block_io_device skip device_block for SES handle(0x0034)
[Sun Jul  9 09:56:16 2023] ses 6:0:34:0: _scsih_block_io_device skip device_block for SES handle(0x0034)
[Sun Jul  9 09:56:16 2023] mpt3sas_cm1: log_info(0x3112010c): originator(PL), code(0x12), sub_code(0x010c)
[Sun Jul  9 09:56:16 2023] mpt3sas_cm1: log_info(0x31120100): originator(PL), code(0x12), sub_code(0x0100)
[Sun Jul  9 09:56:16 2023] mpt3sas_cm1: log_info(0x31120100): originator(PL), code(0x12), sub_code(0x0100)
[Sun Jul  9 09:56:16 2023] mpt3sas_cm1: log_info(0x31120100): originator(PL), code(0x12), sub_code(0x0100)
[Sun Jul  9 09:56:16 2023] mpt3sas_cm1: log_info(0x31120100): originator(PL), code(0x12), sub_code(0x0100)
[Sun Jul  9 09:56:16 2023] mpt3sas_cm1: log_info(0x31120100): originator(PL), code(0x12), sub_code(0x0100)
[Sun Jul  9 09:56:16 2023] mpt3sas_cm1: log_info(0x31120100): originator(PL), code(0x12), sub_code(0x0100)
[Sun Jul  9 09:56:16 2023] mpt3sas_cm1: log_info(0x31120100): originator(PL), code(0x12), sub_code(0x0100)
[Sun Jul  9 09:56:16 2023] mpt3sas_cm1: log_info(0x31120100): originator(PL), code(0x12), sub_code(0x0100)
[Sun Jul  9 09:56:16 2023] mpt3sas_cm1: log_info(0x31120100): originator(PL), code(0x12), sub_code(0x0100)
[Sun Jul  9 09:56:16 2023] mpt3sas_cm1: log_info(0x31120100): originator(PL), code(0x12), sub_code(0x0100)
[Sun Jul  9 09:56:16 2023] mpt3sas_cm1: log_info(0x31120100): originator(PL), code(0x12), sub_code(0x0100)
[Sun Jul  9 09:56:16 2023] mpt3sas_cm1: log_info(0x31120100): originator(PL), code(0x12), sub_code(0x0100)
[Sun Jul  9 09:56:16 2023] mpt3sas_cm1: log_info(0x31120100): originator(PL), code(0x12), sub_code(0x0100)
[Sun Jul  9 09:56:16 2023] mpt3sas_cm1: log_info(0x31120100): originator(PL), code(0x12), sub_code(0x0100)
[Sun Jul  9 09:56:16 2023] mpt3sas_cm1: log_info(0x31120100): originator(PL), code(0x12), sub_code(0x0100)
[Sun Jul  9 09:56:16 2023] mpt3sas_cm1: log_info(0x31120100): originator(PL), code(0x12), sub_code(0x0100)
[Sun Jul  9 09:56:16 2023] ses 0:0:0:0: _scsih_block_io_device skip device_block for SES handle(0x0012)
[Sun Jul  9 09:56:19 2023] ses 0:0:1:0: _scsih_block_io_device skip device_block for SES handle(0x0014)
[Sun Jul  9 09:56:19 2023] sd 6:0:35:0: device_block, handle(0x0036)
[Sun Jul  9 09:56:19 2023] sd 6:0:36:0: device_block, handle(0x0037)
[Sun Jul  9 09:56:19 2023] sd 6:0:37:0: device_block, handle(0x0038)
[Sun Jul  9 09:56:19 2023] sd 6:0:38:0: device_block, handle(0x0039)
[Sun Jul  9 09:56:19 2023] sd 6:0:39:0: device_block, handle(0x003a)
[Sun Jul  9 09:56:19 2023] sd 6:0:40:0: device_block, handle(0x003b)
[Sun Jul  9 09:56:19 2023] sd 6:0:41:0: device_block, handle(0x003c)
[Sun Jul  9 09:56:19 2023] sd 6:0:42:0: device_block, handle(0x003d)
[Sun Jul  9 09:56:19 2023] sd 6:0:43:0: device_block, handle(0x003e)
[Sun Jul  9 09:56:19 2023] sd 6:0:44:0: device_block, handle(0x003f)
[Sun Jul  9 09:56:19 2023] sd 6:0:45:0: device_block, handle(0x0040)
[Sun Jul  9 09:56:19 2023] sd 6:0:46:0: device_block, handle(0x0041)
[Sun Jul  9 09:56:19 2023] sd 6:0:47:0: device_block, handle(0x0042)
[Sun Jul  9 09:56:19 2023] sd 6:0:48:0: device_block, handle(0x0043)
[Sun Jul  9 09:56:19 2023] sd 6:0:49:0: device_block, handle(0x0044)
[Sun Jul  9 09:56:19 2023] sd 6:0:50:0: device_block, handle(0x0045)
[Sun Jul  9 09:56:19 2023] sd 6:0:51:0: device_block, handle(0x0046)
[Sun Jul  9 09:56:19 2023] sd 6:0:52:0: device_block, handle(0x0047)
[Sun Jul  9 09:56:19 2023] sd 6:0:53:0: device_block, handle(0x0048)
[Sun Jul  9 09:56:19 2023] sd 6:0:54:0: device_block, handle(0x0049)
[Sun Jul  9 09:56:19 2023] sd 6:0:55:0: device_block, handle(0x004a)
[Sun Jul  9 09:56:19 2023] sd 6:0:56:0: device_block, handle(0x004b)
[Sun Jul  9 09:56:19 2023] sd 6:0:57:0: device_block, handle(0x004c)
[Sun Jul  9 09:56:19 2023] sd 6:0:58:0: device_block, handle(0x004d)
[Sun Jul  9 09:56:19 2023] sd 6:0:59:0: device_block, handle(0x004e)
[Sun Jul  9 09:56:19 2023] sd 6:0:60:0: device_block, handle(0x004f)
[Sun Jul  9 09:56:19 2023] sd 6:0:61:0: device_block, handle(0x0050)
[Sun Jul  9 09:56:19 2023] sd 6:0:62:0: device_block, handle(0x0051)
[Sun Jul  9 09:56:19 2023] sd 6:0:63:0: device_block, handle(0x0052)
[Sun Jul  9 09:56:19 2023] sd 6:0:64:0: device_block, handle(0x0053)
[Sun Jul  9 09:56:19 2023] sd 6:0:65:0: device_block, handle(0x0054)
[Sun Jul  9 09:56:19 2023] sd 6:0:66:0: device_block, handle(0x0055)
[Sun Jul  9 09:56:19 2023] sd 6:0:67:0: device_block, handle(0x0056)
[Sun Jul  9 09:56:19 2023] sd 6:0:68:0: device_block, handle(0x0057)
[Sun Jul  9 09:56:19 2023] ses 6:0:69:0: _scsih_block_io_device skip device_block for SES handle(0x0058)
[Sun Jul  9 09:56:19 2023] ses 6:0:69:0: _scsih_block_io_device skip device_block for SES handle(0x0058)
[Sun Jul  9 09:56:19 2023] ses 6:0:69:0: _scsih_block_io_device skip device_block for SES handle(0x0058)
[Sun Jul  9 09:56:19 2023] ses 6:0:69:0: _scsih_block_io_device skip device_block for SES handle(0x0058)
[Sun Jul  9 09:56:19 2023] mpt3sas_cm1: log_info(0x3112010c): originator(PL), code(0x12), sub_code(0x010c)
[Sun Jul  9 09:56:19 2023] mpt3sas_cm1: log_info(0x31120100): originator(PL), code(0x12), sub_code(0x0100)
[Sun Jul  9 09:56:19 2023] mpt3sas_cm1: log_info(0x31120100): originator(PL), code(0x12), sub_code(0x0100)
[Sun Jul  9 09:56:19 2023] mpt3sas_cm1: log_info(0x31120100): originator(PL), code(0x12), sub_code(0x0100)
[Sun Jul  9 09:56:19 2023] mpt3sas_cm1: log_info(0x31120100): originator(PL), code(0x12), sub_code(0x0100)
[Sun Jul  9 09:56:19 2023] mpt3sas_cm1: log_info(0x31120100): originator(PL), code(0x12), sub_code(0x0100)
[Sun Jul  9 09:56:20 2023] sd 6:0:0:0: device_unblock and setting to running, handle(0x0012)
[Sun Jul  9 09:56:20 2023] sd 6:0:1:0: device_unblock and setting to running, handle(0x0013)
[Sun Jul  9 09:56:20 2023] sd 6:0:2:0: device_unblock and setting to running, handle(0x0014)
[Sun Jul  9 09:56:20 2023] sd 6:0:3:0: device_unblock and setting to running, handle(0x0015)
[Sun Jul  9 09:56:20 2023] sd 6:0:4:0: device_unblock and setting to running, handle(0x0016)
[Sun Jul  9 09:56:20 2023] sd 6:0:5:0: device_unblock and setting to running, handle(0x0017)
[Sun Jul  9 09:56:20 2023] sd 6:0:6:0: device_unblock and setting to running, handle(0x0018)
[Sun Jul  9 09:56:20 2023] sd 6:0:7:0: device_unblock and setting to running, handle(0x0019)
[Sun Jul  9 09:56:20 2023] sd 6:0:8:0: device_unblock and setting to running, handle(0x001a)
[Sun Jul  9 09:56:20 2023] sd 6:0:9:0: device_unblock and setting to running, handle(0x001b)
[Sun Jul  9 09:56:20 2023] sd 6:0:10:0: device_unblock and setting to running, handle(0x001c)
[Sun Jul  9 09:56:20 2023] sd 6:0:11:0: device_unblock and setting to running, handle(0x001d)
[Sun Jul  9 09:56:20 2023] sd 6:0:12:0: device_unblock and setting to running, handle(0x001e)
[Sun Jul  9 09:56:20 2023] sd 6:0:13:0: device_unblock and setting to running, handle(0x001f)
[Sun Jul  9 09:56:20 2023] sd 6:0:14:0: device_unblock and setting to running, handle(0x0020)
[Sun Jul  9 09:56:20 2023] sd 6:0:15:0: device_unblock and setting to running, handle(0x0021)
[Sun Jul  9 09:56:20 2023] sd 6:0:16:0: device_unblock and setting to running, handle(0x0022)
[Sun Jul  9 09:56:20 2023] sd 6:0:17:0: device_unblock and setting to running, handle(0x0023)
[Sun Jul  9 09:56:20 2023] sd 6:0:18:0: device_unblock and setting to running, handle(0x0024)
[Sun Jul  9 09:56:20 2023] sd 6:0:19:0: device_unblock and setting to running, handle(0x0025)
[Sun Jul  9 09:56:20 2023] sd 6:0:20:0: device_unblock and setting to running, handle(0x0026)
[Sun Jul  9 09:56:20 2023] sd 6:0:21:0: device_unblock and setting to running, handle(0x0027)
[Sun Jul  9 09:56:20 2023] sd 6:0:22:0: device_unblock and setting to running, handle(0x0028)
[Sun Jul  9 09:56:20 2023] sd 6:0:23:0: device_unblock and setting to running, handle(0x0029)
[Sun Jul  9 09:56:20 2023] sd 6:0:24:0: device_unblock and setting to running, handle(0x002a)
[Sun Jul  9 09:56:20 2023] sd 6:0:25:0: device_unblock and setting to running, handle(0x002b)
[Sun Jul  9 09:56:20 2023] sd 6:0:26:0: device_unblock and setting to running, handle(0x002c)
[Sun Jul  9 09:56:20 2023] sd 6:0:27:0: device_unblock and setting to running, handle(0x002d)
[Sun Jul  9 09:56:20 2023] sd 6:0:28:0: device_unblock and setting to running, handle(0x002e)
[Sun Jul  9 09:56:20 2023] sd 6:0:29:0: device_unblock and setting to running, handle(0x002f)
[Sun Jul  9 09:56:20 2023] sd 6:0:30:0: device_unblock and setting to running, handle(0x0030)
[Sun Jul  9 09:56:20 2023] sd 6:0:31:0: device_unblock and setting to running, handle(0x0031)
[Sun Jul  9 09:56:20 2023] sd 6:0:32:0: device_unblock and setting to running, handle(0x0032)
[Sun Jul  9 09:56:20 2023] sd 6:0:33:0: device_unblock and setting to running, handle(0x0033)
[Sun Jul  9 09:56:20 2023] mpt3sas_cm1: mpt3sas_transport_port_remove: removed: sas_addr(0x5000c500cac022ca)
[Sun Jul  9 09:56:20 2023] mpt3sas_cm1: removing handle(0x0012), sas_addr(0x5000c500cac022ca)
[Sun Jul  9 09:56:20 2023] mpt3sas_cm1: enclosure logical id(0x540f078d86014000), slot(1)
[Sun Jul  9 09:56:20 2023] mpt3sas_cm1: enclosure level(0x0000), connector name(     )
[Sun Jul  9 09:56:20 2023] mpt3sas_cm1: log_info(0x31130000): originator(PL), code(0x13), sub_code(0x0000)
[Sun Jul  9 09:56:20 2023] mpt3sas_cm1: log_info(0x31130000): originator(PL), code(0x13), sub_code(0x0000)
[Sun Jul  9 09:56:20 2023] mpt3sas_cm1: log_info(0x31130000): originator(PL), code(0x13), sub_code(0x0000)
[Sun Jul  9 09:56:20 2023] mpt3sas_cm1: log_info(0x31130000): originator(PL), code(0x13), sub_code(0x0000)
[Sun Jul  9 09:56:20 2023] mpt3sas_cm1: log_info(0x31130000): originator(PL), code(0x13), sub_code(0x0000)
[Sun Jul  9 09:56:20 2023] mpt3sas_cm1: log_info(0x31130000): originator(PL), code(0x13), sub_code(0x0000)
[Sun Jul  9 09:56:20 2023] mpt3sas_cm1: log_info(0x31130000): originator(PL), code(0x13), sub_code(0x0000)
[Sun Jul  9 09:56:20 2023] mpt3sas_cm1: log_info(0x31130000): originator(PL), code(0x13), sub_code(0x0000)
[Sun Jul  9 09:56:20 2023] mpt3sas_cm1: log_info(0x31130000): originator(PL), code(0x13), sub_code(0x0000)
[Sun Jul  9 09:56:20 2023] mpt3sas_cm1: log_info(0x31130000): originator(PL), code(0x13), sub_code(0x0000)
[Sun Jul  9 09:56:20 2023] mpt3sas_cm1: log_info(0x31130000): originator(PL), code(0x13), sub_code(0x0000)
[Sun Jul  9 09:56:20 2023] mpt3sas_cm1: log_info(0x31130000): originator(PL), code(0x13), sub_code(0x0000)
[Sun Jul  9 09:56:20 2023] mpt3sas_cm1: log_info(0x31130000): originator(PL), code(0x13), sub_code(0x0000)
[Sun Jul  9 09:56:20 2023] mpt3sas_cm1: log_info(0x31130000): originator(PL), code(0x13), sub_code(0x0000)
[Sun Jul  9 09:56:20 2023] mpt3sas_cm1: log_info(0x31130000): originator(PL), code(0x13), sub_code(0x0000)
[Sun Jul  9 09:56:20 2023] mpt3sas_cm1: log_info(0x31130000): originator(PL), code(0x13), sub_code(0x0000)
[Sun Jul  9 09:56:20 2023] mpt3sas_cm1: log_info(0x31130000): originator(PL), code(0x13), sub_code(0x0000)
[Sun Jul  9 09:56:20 2023] mpt3sas_cm1: log_info(0x31130000): originator(PL), code(0x13), sub_code(0x0000)
[Sun Jul  9 09:56:20 2023] mpt3sas_cm1: log_info(0x31130000): originator(PL), code(0x13), sub_code(0x0000)
[Sun Jul  9 09:56:20 2023] mpt3sas_cm1: log_info(0x31130000): originator(PL), code(0x13), sub_code(0x0000)
[Sun Jul  9 09:56:20 2023] mpt3sas_cm1: log_info(0x31130000): originator(PL), code(0x13), sub_code(0x0000)
[Sun Jul  9 09:56:20 2023] mpt3sas_cm1: mpt3sas_transport_port_remove: removed: sas_addr(0x5000c500cabfed12)
[Sun Jul  9 09:56:20 2023] mpt3sas_cm1: removing handle(0x0013), sas_addr(0x5000c500cabfed12)
[Sun Jul  9 09:56:20 2023] mpt3sas_cm1: enclosure logical id(0x540f078d86014000), slot(2)
[Sun Jul  9 09:56:20 2023] mpt3sas_cm1: enclosure level(0x0000), connector name(     )
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/03453b32-5793-4113-9ebd-64723a51b017 error=5 type=2 offset=8594128896 size=4096 flags=180880
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/03453b32-5793-4113-9ebd-64723a51b017 error=5 type=2 offset=9821274112 size=4096 flags=180880
[Sun Jul  9 09:56:20 2023] sd 6:0:5:0: [sdh] tag#4609 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=0s
[Sun Jul  9 09:56:20 2023] sd 6:0:5:0: [sdh] tag#4609 CDB: Write(16) 8a 00 00 00 00 00 01 6d d7 08 00 00 00 08 00 00
[Sun Jul  9 09:56:20 2023] blk_update_request: I/O error, dev sdh, sector 23975688 op 0x1:(WRITE) flags 0x700 phys_seg 1 prio class 0
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/91a2cc2b-bc27-47f6-b127-018d41645a67 error=5 type=2 offset=12275531776 size=4096 flags=180880
[Sun Jul  9 09:56:20 2023] sd 6:0:22:0: [sdy] tag#2963 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=0s
[Sun Jul  9 09:56:20 2023] sd 6:0:22:0: [sdy] tag#2963 CDB: Write(16) 8a 00 00 00 00 00 01 92 69 58 00 00 00 08 00 00
[Sun Jul  9 09:56:20 2023] blk_update_request: I/O error, dev sdy, sector 26372440 op 0x1:(WRITE) flags 0x700 phys_seg 1 prio class 0
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/2a2364c1-5273-46ad-9f85-e27ca227ee9a error=5 type=2 offset=13502668800 size=4096 flags=180880
[Sun Jul  9 09:56:20 2023] sd 6:0:19:0: [sdv] tag#4610 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=0s
[Sun Jul  9 09:56:20 2023] sd 6:0:19:0: [sdv] tag#4610 CDB: Write(16) 8a 00 00 00 00 00 01 6d d7 08 00 00 00 08 00 00
[Sun Jul  9 09:56:20 2023] blk_update_request: I/O error, dev sdv, sector 23975688 op 0x1:(WRITE) flags 0x700 phys_seg 1 prio class 0
[Sun Jul  9 09:56:20 2023] sd 6:0:14:0: [sds] tag#2964 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=0s
[Sun Jul  9 09:56:20 2023] sd 6:0:5:0: [sdh] tag#634 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=0s
[Sun Jul  9 09:56:20 2023] sd 6:0:14:0: [sds] tag#2964 CDB: Write(16) 8a 00 00 00 00 00 01 92 69 60 00 00 00 08 00 00
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/9c8f9597-9817-476d-b063-b8483646781b error=5 type=2 offset=12275531776 size=4096 flags=180880
[Sun Jul  9 09:56:20 2023] blk_update_request: I/O error, dev sds, sector 26372448 op 0x1:(WRITE) flags 0x700 phys_seg 1 prio class 0
[Sun Jul  9 09:56:20 2023] sd 6:0:5:0: [sdh] tag#634 CDB: Read(16) 88 00 00 00 00 00 00 00 02 38 00 00 00 10 00 00
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/574761ca-a585-4a9d-ac2f-a9e8051ba5dd error=5 type=2 offset=13502672896 size=4096 flags=180880
[Sun Jul  9 09:56:20 2023] blk_update_request: I/O error, dev sdh, sector 568 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[Sun Jul  9 09:56:20 2023] sd 6:0:22:0: [sdy] tag#1006 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=0s
[Sun Jul  9 09:56:20 2023] sd 6:0:22:0: [sdy] tag#1006 CDB: Read(16) 88 00 00 00 00 00 00 00 02 38 00 00 00 10 00 00
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/91a2cc2b-bc27-47f6-b127-018d41645a67 error=5 type=1 offset=270336 size=8192 flags=b08c1
[Sun Jul  9 09:56:20 2023] blk_update_request: I/O error, dev sdy, sector 568 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/2a2364c1-5273-46ad-9f85-e27ca227ee9a error=5 type=1 offset=270336 size=8192 flags=b08c1
[Sun Jul  9 09:56:20 2023] sd 6:0:5:0: [sdh] tag#635 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=0s
[Sun Jul  9 09:56:20 2023] sd 6:0:22:0: [sdy] tag#1007 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=0s
[Sun Jul  9 09:56:20 2023] sd 6:0:22:0: [sdy] tag#1007 CDB: Read(16) 88 00 00 00 00 01 d1 c0 ba 38 00 00 00 10 00 00
[Sun Jul  9 09:56:20 2023] sd 6:0:5:0: [sdh] tag#635 CDB: Read(16) 88 00 00 00 00 01 d1 c0 ba 38 00 00 00 10 00 00
[Sun Jul  9 09:56:20 2023] blk_update_request: I/O error, dev sdy, sector 7814036024 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[Sun Jul  9 09:56:20 2023] sd 6:0:11:0: [sdn] tag#4611 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=0s
[Sun Jul  9 09:56:20 2023] sd 6:0:15:0: [sdp] tag#2965 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=0s
[Sun Jul  9 09:56:20 2023] blk_update_request: I/O error, dev sdh, sector 7814036024 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/2a2364c1-5273-46ad-9f85-e27ca227ee9a error=5 type=1 offset=4000786423808 size=8192 flags=b08c1
[Sun Jul  9 09:56:20 2023] sd 6:0:15:0: [sdp] tag#2965 CDB: Write(16) 8a 00 00 00 00 00 01 92 69 60 00 00 00 08 00 00
[Sun Jul  9 09:56:20 2023] sd 6:0:11:0: [sdn] tag#4611 CDB: Write(16) 8a 00 00 00 00 00 01 92 69 50 00 00 00 08 00 00
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/91a2cc2b-bc27-47f6-b127-018d41645a67 error=5 type=1 offset=4000786423808 size=8192 flags=b08c1

[Sun Jul  9 09:56:20 2023] blk_update_request: I/O error, dev sdp, sector 26372448 op 0x1:(WRITE) flags 0x700 phys_seg 1 prio class 0
[Sun Jul  9 09:56:20 2023] blk_update_request: I/O error, dev sdn, sector 26372432 op 0x1:(WRITE) flags 0x700 phys_seg 1 prio class 0

mpt3sas_cm1 starts removing all of my SAS_HDD drives one by one but do not touches the SAS_SSD drives.

[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/2a2364c1-5273-46ad-9f85-e27ca227ee9a error=5 type=1 offset=4000786685952 size=8192 flags=b08c1
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/60bee690-1b9c-41e2-9ad7-c94a72411212 error=5 type=2 offset=13502672896 size=4096 flags=180880
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/a0b87b9c-2e68-4f74-ae61-f0250f2444ec error=5 type=2 offset=13502664704 size=4096 flags=180880
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/91a2cc2b-bc27-47f6-b127-018d41645a67 error=5 type=1 offset=4000786685952 size=8192 flags=b08c1
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/4422dcf4-3dc2-41d4-800e-aaa3b1b4c5c5 error=5 type=2 offset=13502664704 size=4096 flags=180880
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/574761ca-a585-4a9d-ac2f-a9e8051ba5dd error=5 type=1 offset=270336 size=8192 flags=b08c1
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/9c8f9597-9817-476d-b063-b8483646781b error=5 type=1 offset=270336 size=8192 flags=b08c1
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/36f31d95-9e20-4f0e-9717-986a784a6b65 error=5 type=2 offset=14729805824 size=4096 flags=180880
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/574761ca-a585-4a9d-ac2f-a9e8051ba5dd error=5 type=1 offset=4000786423808 size=8192 flags=b08c1
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/9c8f9597-9817-476d-b063-b8483646781b error=5 type=1 offset=4000786423808 size=8192 flags=b08c1
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/574761ca-a585-4a9d-ac2f-a9e8051ba5dd error=5 type=1 offset=4000786685952 size=8192 flags=b08c1
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/3dfd5d37-be5f-4d0f-92ae-74aa137e4cd4 error=5 type=2 offset=14729805824 size=4096 flags=180880
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/9c8f9597-9817-476d-b063-b8483646781b error=5 type=1 offset=4000786685952 size=8192 flags=b08c1
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/60bee690-1b9c-41e2-9ad7-c94a72411212 error=5 type=2 offset=14729805824 size=4096 flags=180880
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/36f31d95-9e20-4f0e-9717-986a784a6b65 error=5 type=1 offset=270336 size=8192 flags=b08c1
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/4422dcf4-3dc2-41d4-800e-aaa3b1b4c5c5 error=5 type=1 offset=270336 size=8192 flags=b08c1
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/a0b87b9c-2e68-4f74-ae61-f0250f2444ec error=5 type=1 offset=270336 size=8192 flags=b08c1
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/36f31d95-9e20-4f0e-9717-986a784a6b65 error=5 type=1 offset=4000786423808 size=8192 flags=b08c1
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/4422dcf4-3dc2-41d4-800e-aaa3b1b4c5c5 error=5 type=1 offset=4000786423808 size=8192 flags=b08c1
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/36f31d95-9e20-4f0e-9717-986a784a6b65 error=5 type=1 offset=4000786685952 size=8192 flags=b08c1
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/4422dcf4-3dc2-41d4-800e-aaa3b1b4c5c5 error=5 type=1 offset=4000786685952 size=8192 flags=b08c1
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/cd04703b-d694-47bc-88a4-add49c01907c error=5 type=2 offset=14729805824 size=4096 flags=180880
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/a0b87b9c-2e68-4f74-ae61-f0250f2444ec error=5 type=1 offset=4000786423808 size=8192 flags=b08c1
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/60bee690-1b9c-41e2-9ad7-c94a72411212 error=5 type=1 offset=270336 size=8192 flags=b08c1
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/a0b87b9c-2e68-4f74-ae61-f0250f2444ec error=5 type=1 offset=4000786685952 size=8192 flags=b08c1
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/404c0baa-9f40-45b4-984d-92c75e03b94d error=5 type=2 offset=14729805824 size=4096 flags=180880
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/60bee690-1b9c-41e2-9ad7-c94a72411212 error=5 type=1 offset=4000786423808 size=8192 flags=b08c1
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/60bee690-1b9c-41e2-9ad7-c94a72411212 error=5 type=1 offset=4000786685952 size=8192 flags=b08c1
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/3dfd5d37-be5f-4d0f-92ae-74aa137e4cd4 error=5 type=1 offset=270336 size=8192 flags=b08c1
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/3dfd5d37-be5f-4d0f-92ae-74aa137e4cd4 error=5 type=1 offset=4000786423808 size=8192 flags=b08c1
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/3dfd5d37-be5f-4d0f-92ae-74aa137e4cd4 error=5 type=1 offset=4000786685952 size=8192 flags=b08c1
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/cd04703b-d694-47bc-88a4-add49c01907c error=5 type=1 offset=270336 size=8192 flags=b08c1
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/cd04703b-d694-47bc-88a4-add49c01907c error=5 type=1 offset=4000786423808 size=8192 flags=b08c1
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/cd04703b-d694-47bc-88a4-add49c01907c error=5 type=1 offset=4000786685952 size=8192 flags=b08c1
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/404c0baa-9f40-45b4-984d-92c75e03b94d error=5 type=1 offset=270336 size=8192 flags=b08c1
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/404c0baa-9f40-45b4-984d-92c75e03b94d error=5 type=1 offset=4000786423808 size=8192 flags=b08c1
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/404c0baa-9f40-45b4-984d-92c75e03b94d error=5 type=1 offset=4000786685952 size=8192 flags=b08c1
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/03453b32-5793-4113-9ebd-64723a51b017 error=5 type=2 offset=12275531776 size=4096 flags=180880
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/d115d997-0c9b-4031-8df5-b0c80ae4593e error=5 type=2 offset=8594128896 size=4096 flags=180880
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/91a2cc2b-bc27-47f6-b127-018d41645a67 error=5 type=2 offset=8594128896 size=4096 flags=180880
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/d115d997-0c9b-4031-8df5-b0c80ae4593e error=5 type=1 offset=270336 size=8192 flags=b08c1
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/d115d997-0c9b-4031-8df5-b0c80ae4593e error=5 type=1 offset=4000786423808 size=8192 flags=b08c1
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/d115d997-0c9b-4031-8df5-b0c80ae4593e error=5 type=1 offset=4000786685952 size=8192 flags=b08c1
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/f4a6e721-b119-4476-928f-24eaaa9d7af6 error=5 type=2 offset=11048407040 size=4096 flags=180880
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/64bb4e1d-23e7-47be-ae78-42ff30dddb47 error=5 type=2 offset=11048407040 size=4096 flags=180880
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/64bb4e1d-23e7-47be-ae78-42ff30dddb47 error=5 type=1 offset=270336 size=8192 flags=b08c1
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/64bb4e1d-23e7-47be-ae78-42ff30dddb47 error=5 type=1 offset=4000786423808 size=8192 flags=b08c1
[Sun Jul  9 09:56:20 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/64bb4e1d-23e7-47be-ae78-42ff30dddb47 error=5 type=1 offset=4000786685952 size=8192 flags=b08c1
[Sun Jul  9 09:56:21 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/f4a6e721-b119-4476-928f-24eaaa9d7af6 error=5 type=1 offset=270336 size=8192 flags=b08c1
[Sun Jul  9 09:56:21 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/f4a6e721-b119-4476-928f-24eaaa9d7af6 error=5 type=1 offset=4000786423808 size=8192 flags=b08c1
[Sun Jul  9 09:56:21 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/f4a6e721-b119-4476-928f-24eaaa9d7af6 error=5 type=1 offset=4000786685952 size=8192 flags=b08c1
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: mpt3sas_transport_port_remove: removed: sas_addr(0x5000c500cabfc96a)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: removing handle(0x0014), sas_addr(0x5000c500cabfc96a)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: enclosure logical id(0x540f078d86014000), slot(3)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: enclosure level(0x0000), connector name(     )
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: mpt3sas_transport_port_remove: removed: sas_addr(0x5000c500cac03472)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: removing handle(0x0015), sas_addr(0x5000c500cac03472)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: enclosure logical id(0x540f078d86014000), slot(4)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: enclosure level(0x0000), connector name(     )
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: mpt3sas_transport_port_remove: removed: sas_addr(0x5000c500cac03916)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: removing handle(0x0016), sas_addr(0x5000c500cac03916)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: enclosure logical id(0x540f078d86014000), slot(5)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: enclosure level(0x0000), connector name(     )
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: mpt3sas_transport_port_remove: removed: sas_addr(0x5000c500cabfcdd6)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: removing handle(0x0017), sas_addr(0x5000c500cabfcdd6)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: enclosure logical id(0x540f078d86014000), slot(6)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: enclosure level(0x0000), connector name(     )
[Sun Jul  9 09:56:21 2023] mpt3sas_cm0: log_info(0x31130000): originator(PL), code(0x13), sub_code(0x0000)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm0: mpt3sas_transport_port_remove: removed: sas_addr(0x540f078d86015b7e)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm0: removing handle(0x0012), sas_addr(0x540f078d86015b7e)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm0: enclosure logical id(0x540f078d86014000), slot(61)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm0: enclosure level(0x0000), connector name(     )
[Sun Jul  9 09:56:21 2023] mpt3sas_cm0: mpt3sas_transport_port_remove: removed: sas_addr(0x540f078d86015b7f)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: mpt3sas_transport_port_remove: removed: sas_addr(0x5000c500cac044c2)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm0: expander_remove: handle(0x0011), sas_addr(0x540f078d86015b7f), port:255
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: removing handle(0x0018), sas_addr(0x5000c500cac044c2)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: enclosure logical id(0x540f078d86014000), slot(7)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: enclosure level(0x0000), connector name(     )
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: mpt3sas_transport_port_remove: removed: sas_addr(0x5000c500cac0402a)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: removing handle(0x0019), sas_addr(0x5000c500cac0402a)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: enclosure logical id(0x540f078d86014000), slot(8)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: enclosure level(0x0000), connector name(     )
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: mpt3sas_transport_port_remove: removed: sas_addr(0x5000c500cabfe52a)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: removing handle(0x001a), sas_addr(0x5000c500cabfe52a)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: enclosure logical id(0x540f078d86014000), slot(9)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: enclosure level(0x0000), connector name(     )
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: mpt3sas_transport_port_remove: removed: sas_addr(0x5000c500cabfc366)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: removing handle(0x001b), sas_addr(0x5000c500cabfc366)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: enclosure logical id(0x540f078d86014000), slot(10)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: enclosure level(0x0000), connector name(     )
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: mpt3sas_transport_port_remove: removed: sas_addr(0x5000c500cabfde1a)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: removing handle(0x001c), sas_addr(0x5000c500cabfde1a)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: enclosure logical id(0x540f078d86014000), slot(11)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: enclosure level(0x0000), connector name(     )
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: mpt3sas_transport_port_remove: removed: sas_addr(0x5000c500cabffdee)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: removing handle(0x001d), sas_addr(0x5000c500cabffdee)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: enclosure logical id(0x540f078d86014000), slot(12)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: enclosure level(0x0000), connector name(     )
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: mpt3sas_transport_port_remove: removed: sas_addr(0x5000c500cac00e76)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: removing handle(0x001e), sas_addr(0x5000c500cac00e76)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: enclosure logical id(0x540f078d86014000), slot(13)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: enclosure level(0x0000), connector name(     )
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: mpt3sas_transport_port_remove: removed: sas_addr(0x5000c500cabfc7a6)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: removing handle(0x001f), sas_addr(0x5000c500cabfc7a6)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: enclosure logical id(0x540f078d86014000), slot(14)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: enclosure level(0x0000), connector name(     )
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: mpt3sas_transport_port_remove: removed: sas_addr(0x5000c500cabfdf0a)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: removing handle(0x0020), sas_addr(0x5000c500cabfdf0a)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: enclosure logical id(0x540f078d86014000), slot(15)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: enclosure level(0x0000), connector name(     )
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: mpt3sas_transport_port_remove: removed: sas_addr(0x5000c500cabfb0ee)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: removing handle(0x0021), sas_addr(0x5000c500cabfb0ee)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: enclosure logical id(0x540f078d86014000), slot(16)
[Sun Jul  9 09:56:21 2023] mpt3sas_cm1: enclosure level(0x0000), connector name(     )
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: mpt3sas_transport_port_remove: removed: sas_addr(0x5000c500cabfcd36)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: removing handle(0x0022), sas_addr(0x5000c500cabfcd36)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: enclosure logical id(0x540f078d86014000), slot(17)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: enclosure level(0x0000), connector name(     )
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: mpt3sas_transport_port_remove: removed: sas_addr(0x5000c500cabfd5ee)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: removing handle(0x0023), sas_addr(0x5000c500cabfd5ee)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: enclosure logical id(0x540f078d86014000), slot(18)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: enclosure level(0x0000), connector name(     )
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: mpt3sas_transport_port_remove: removed: sas_addr(0x5000c500cabfd262)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: removing handle(0x0024), sas_addr(0x5000c500cabfd262)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: enclosure logical id(0x540f078d86014000), slot(19)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: enclosure level(0x0000), connector name(     )
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: mpt3sas_transport_port_remove: removed: sas_addr(0x5000c500cabfc2c6)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: removing handle(0x0025), sas_addr(0x5000c500cabfc2c6)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: enclosure logical id(0x540f078d86014000), slot(20)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: enclosure level(0x0000), connector name(     )
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: mpt3sas_transport_port_remove: removed: sas_addr(0x5000c500cabfe1ee)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: removing handle(0x0026), sas_addr(0x5000c500cabfe1ee)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: enclosure logical id(0x540f078d86014000), slot(21)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: enclosure level(0x0000), connector name(     )
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: mpt3sas_transport_port_remove: removed: sas_addr(0x5000c500cabfb4ea)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: removing handle(0x0027), sas_addr(0x5000c500cabfb4ea)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: enclosure logical id(0x540f078d86014000), slot(22)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: enclosure level(0x0000), connector name(     )
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: mpt3sas_transport_port_remove: removed: sas_addr(0x5000c500cabfceee)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: removing handle(0x0028), sas_addr(0x5000c500cabfceee)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: enclosure logical id(0x540f078d86014000), slot(23)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: enclosure level(0x0000), connector name(     )
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: mpt3sas_transport_port_remove: removed: sas_addr(0x5000c500cabfca36)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: removing handle(0x0029), sas_addr(0x5000c500cabfca36)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: enclosure logical id(0x540f078d86014000), slot(24)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: enclosure level(0x0000), connector name(     )
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: mpt3sas_transport_port_remove: removed: sas_addr(0x5000c500cabfbb96)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: removing handle(0x002a), sas_addr(0x5000c500cabfbb96)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: enclosure logical id(0x540f078d86014000), slot(25)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: enclosure level(0x0000), connector name(     )
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: mpt3sas_transport_port_remove: removed: sas_addr(0x5000c500cac03dd2)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: removing handle(0x002b), sas_addr(0x5000c500cac03dd2)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: enclosure logical id(0x540f078d86014000), slot(26)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: enclosure level(0x0000), connector name(     )
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: mpt3sas_transport_port_remove: removed: sas_addr(0x5000c500cabfcd1e)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: removing handle(0x002c), sas_addr(0x5000c500cabfcd1e)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: enclosure logical id(0x540f078d86014000), slot(27)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: enclosure level(0x0000), connector name(     )
[Sun Jul  9 09:56:22 2023] mpt3sas_cm0: expander_add: handle(0x0011), parent(0x0009), sas_addr(0x540f078d86015b7f), phys(69)
[Sun Jul  9 09:56:22 2023]  expander-0:2: add: handle(0x0011), sas_addr(0x540f078d86015b7f)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm0: handle(0x12) sas_address(0x540f078d86015b7e) port_type(0x1)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: mpt3sas_transport_port_remove: removed: sas_addr(0x5000c500cabfcde2)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: removing handle(0x002d), sas_addr(0x5000c500cabfcde2)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: enclosure logical id(0x540f078d86014000), slot(28)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: enclosure level(0x0000), connector name(     )
[Sun Jul  9 09:56:22 2023] sd 6:0:28:0: [sdae] Synchronizing SCSI cache
[Sun Jul  9 09:56:22 2023] sd 6:0:28:0: [sdae] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: mpt3sas_transport_port_remove: removed: sas_addr(0x5002538a072e70b3)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: removing handle(0x002e), sas_addr(0x5002538a072e70b3)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: enclosure logical id(0x540f078d86014000), slot(51)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: enclosure level(0x0000), connector name(     )
[Sun Jul  9 09:56:22 2023] sd 6:0:29:0: [sdaf] Synchronizing SCSI cache
[Sun Jul  9 09:56:22 2023] sd 6:0:29:0: [sdaf] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: mpt3sas_transport_port_remove: removed: sas_addr(0x5002538a072ec943)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: removing handle(0x002f), sas_addr(0x5002538a072ec943)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: enclosure logical id(0x540f078d86014000), slot(52)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: enclosure level(0x0000), connector name(     )
[Sun Jul  9 09:56:22 2023] sd 6:0:30:0: [sdag] Synchronizing SCSI cache
[Sun Jul  9 09:56:22 2023] sd 6:0:30:0: [sdag] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: mpt3sas_transport_port_remove: removed: sas_addr(0x5002538a072f0133)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: removing handle(0x0030), sas_addr(0x5002538a072f0133)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: enclosure logical id(0x540f078d86014000), slot(53)
[Sun Jul  9 09:56:22 2023] mpt3sas_cm1: enclosure level(0x0000), connector name(     )
[Sun Jul  9 09:56:23 2023] sd 6:0:31:0: [sdah] Synchronizing SCSI cache
[Sun Jul  9 09:56:23 2023] sd 6:0:31:0: [sdah] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
[Sun Jul  9 09:56:23 2023] mpt3sas_cm1: mpt3sas_transport_port_remove: removed: sas_addr(0x5002538a072dfe73)
[Sun Jul  9 09:56:23 2023] mpt3sas_cm1: removing handle(0x0031), sas_addr(0x5002538a072dfe73)
[Sun Jul  9 09:56:23 2023] mpt3sas_cm1: enclosure logical id(0x540f078d86014000), slot(54)
[Sun Jul  9 09:56:23 2023] mpt3sas_cm1: enclosure level(0x0000), connector name(     )
[Sun Jul  9 09:56:23 2023] sd 6:0:32:0: [sdai] Synchronizing SCSI cache
[Sun Jul  9 09:56:23 2023] sd 6:0:32:0: [sdai] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
[Sun Jul  9 09:56:23 2023] mpt3sas_cm1: mpt3sas_transport_port_remove: removed: sas_addr(0x5002538a072df683)
[Sun Jul  9 09:56:23 2023] mpt3sas_cm1: removing handle(0x0032), sas_addr(0x5002538a072df683)
[Sun Jul  9 09:56:23 2023] mpt3sas_cm1: enclosure logical id(0x540f078d86014000), slot(55)
[Sun Jul  9 09:56:23 2023] mpt3sas_cm1: enclosure level(0x0000), connector name(     )
[Sun Jul  9 09:56:23 2023] sd 6:0:33:0: [sdaj] Synchronizing SCSI cache
[Sun Jul  9 09:56:23 2023] sd 6:0:33:0: [sdaj] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
[Sun Jul  9 09:56:23 2023] mpt3sas_cm1: mpt3sas_transport_port_remove: removed: sas_addr(0x5002538a072e0003)
[Sun Jul  9 09:56:23 2023] mpt3sas_cm1: removing handle(0x0033), sas_addr(0x5002538a072e0003)
[Sun Jul  9 09:56:23 2023] mpt3sas_cm1: enclosure logical id(0x540f078d86014000), slot(56)
[Sun Jul  9 09:56:23 2023] mpt3sas_cm1: enclosure level(0x0000), connector name(     )
[Sun Jul  9 09:56:23 2023] mpt3sas_cm1: mpt3sas_transport_port_remove: removed: sas_addr(0x540f078d86015b7e)
[Sun Jul  9 09:56:23 2023] mpt3sas_cm1: removing handle(0x0034), sas_addr(0x540f078d86015b7e)
[Sun Jul  9 09:56:23 2023] mpt3sas_cm1: enclosure logical id(0x540f078d86014000), slot(61)
[Sun Jul  9 09:56:23 2023] mpt3sas_cm1: enclosure level(0x0000), connector name(     )
[Sun Jul  9 09:56:23 2023] mpt3sas_cm1: mpt3sas_transport_port_remove: removed: sas_addr(0x540f078d86015b7f)
[Sun Jul  9 09:56:23 2023] mpt3sas_cm1: expander_remove: handle(0x0011), sas_addr(0x540f078d86015b7f), port:255
[Sun Jul  9 09:56:23 2023] mpt3sas_cm1: expander_add: handle(0x0011), parent(0x0009), sas_addr(0x540f078d86015b7f), phys(69)
[Sun Jul  9 09:56:23 2023]  expander-6:2: add: handle(0x0011), sas_addr(0x540f078d86015b7f)
[Sun Jul  9 09:56:23 2023] mpt3sas_cm1: handle(0x34) sas_address(0x540f078d86015b7e) port_type(0x1)
[Sun Jul  9 09:56:23 2023] scsi 6:0:70:0: Enclosure         Cisco    C3260            2    PQ: 0 ANSI: 6
[Sun Jul  9 09:56:23 2023] scsi 6:0:70:0: set ignore_delay_remove for handle(0x0034)
[Sun Jul  9 09:56:23 2023] scsi 6:0:70:0: SES: handle(0x0034), sas_addr(0x540f078d86015b7e), phy(68), device_name(0x0000000000000000)
[Sun Jul  9 09:56:23 2023] scsi 6:0:70:0: enclosure logical id (0x540f078d86014000), slot(61) 
[Sun Jul  9 09:56:23 2023] scsi 6:0:70:0: enclosure level(0x0000), connector name(     )
[Sun Jul  9 09:56:23 2023] scsi 6:0:70:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
[Sun Jul  9 09:56:23 2023] scsi 6:0:70:0: Power-on or device reset occurred
[Sun Jul  9 09:56:23 2023] ses 6:0:70:0: Attached Enclosure device
[Sun Jul  9 09:56:23 2023] ses 6:0:70:0: Attached scsi generic sg3 type 13
[Sun Jul  9 09:56:23 2023]  end_device-6:2:0: add: handle(0x0034), sas_addr(0x540f078d86015b7e)
[Sun Jul  9 09:56:23 2023] mpt3sas_cm1: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05)
[Sun Jul  9 09:56:23 2023] ses 6:0:70:0: Power-on or device reset occurred
[Sun Jul  9 09:56:24 2023] mpt3sas_cm0: log_info(0x31130000): originator(PL), code(0x13), sub_code(0x0000)
[Sun Jul  9 09:56:24 2023] scsi 0:0:2:0: Enclosure         Cisco    C3260            2    PQ: 0 ANSI: 6
[Sun Jul  9 09:56:24 2023] scsi 0:0:2:0: set ignore_delay_remove for handle(0x0012)
[Sun Jul  9 09:56:24 2023] scsi 0:0:2:0: SES: handle(0x0012), sas_addr(0x540f078d86015b7e), phy(68), device_name(0x0000000000000000)
[Sun Jul  9 09:56:24 2023] scsi 0:0:2:0: enclosure logical id (0x540f078d86014000), slot(61) 
[Sun Jul  9 09:56:24 2023] scsi 0:0:2:0: enclosure level(0x0000), connector name(     )
[Sun Jul  9 09:56:24 2023] scsi 0:0:2:0: qdepth(254), tagged(1), scsi_level(7), cmd_que(1)
[Sun Jul  9 09:56:24 2023] scsi 0:0:2:0: Power-on or device reset occurred
[Sun Jul  9 09:56:24 2023] ses 0:0:2:0: Attached Enclosure device
[Sun Jul  9 09:56:24 2023] ses 0:0:2:0: Attached scsi generic sg5 type 13
[Sun Jul  9 09:56:24 2023]  end_device-0:2:0: add: handle(0x0012), sas_addr(0x540f078d86015b7e)
[Sun Jul  9 09:56:24 2023] mpt3sas_cm0: mpt3sas_transport_port_remove: removed: sas_addr(0x540f078d86014a7e)
[Sun Jul  9 09:56:24 2023] mpt3sas_cm0: removing handle(0x0014), sas_addr(0x540f078d86014a7e)
[Sun Jul  9 09:56:24 2023] mpt3sas_cm0: enclosure logical id(0x540f078d86014000), slot(61)
[Sun Jul  9 09:56:24 2023] mpt3sas_cm0: enclosure level(0x0000), connector name(     )
[Sun Jul  9 09:56:24 2023] mpt3sas_cm0: mpt3sas_transport_port_remove: removed: sas_addr(0x540f078d86014a7f)
[Sun Jul  9 09:56:24 2023] mpt3sas_cm0: expander_remove: handle(0x0013), sas_addr(0x540f078d86014a7f), port:255
[Sun Jul  9 09:56:24 2023] mpt3sas_cm1: handle(0x12) sas_address(0x5000c500cac022ca) port_type(0x1)
[Sun Jul  9 09:56:24 2023] scsi 6:0:71:0: Direct-Access     SEAGATE  ST4000NM017A     CN04 PQ: 0 ANSI: 7
[Sun Jul  9 09:56:24 2023] scsi 6:0:71:0: SSP: handle(0x0012), sas_addr(0x5000c500cac022ca), phy(0), device_name(0x5000c500cac022c8)
[Sun Jul  9 09:56:24 2023] scsi 6:0:71:0: enclosure logical id (0x540f078d86014000), slot(1) 
[Sun Jul  9 09:56:24 2023] scsi 6:0:71:0: enclosure level(0x0000), connector name(     )
[Sun Jul  9 09:56:24 2023] scsi 6:0:71:0: qdepth(254), tagged(1), scsi_level(8), cmd_que(1)
[Sun Jul  9 09:56:24 2023] sd 6:0:71:0: [sdc] 7814037168 512-byte logical blocks: (4.00 TB/3.64 TiB)
[Sun Jul  9 09:56:24 2023] sd 6:0:71:0: [sdc] Write Protect is off
[Sun Jul  9 09:56:24 2023] sd 6:0:71:0: [sdc] Mode Sense: df 00 10 08
[Sun Jul  9 09:56:24 2023] sd 6:0:71:0: [sdc] Write cache: disabled, read cache: enabled, supports DPO and FUA
[Sun Jul  9 09:56:24 2023] debugfs: Directory 'sdc' with parent 'block' already present!
[Sun Jul  9 09:56:24 2023]  sdc: sdc1
[Sun Jul  9 09:56:24 2023] sd 6:0:71:0: [sdc] Attached SCSI disk
[Sun Jul  9 09:56:24 2023] sd 6:0:35:0: device_unblock and setting to running, handle(0x0036)
[Sun Jul  9 09:56:24 2023] sd 6:0:36:0: device_unblock and setting to running, handle(0x0037)
[Sun Jul  9 09:56:24 2023] sd 6:0:37:0: device_unblock and setting to running, handle(0x0038)
[Sun Jul  9 09:56:24 2023] sd 6:0:38:0: device_unblock and setting to running, handle(0x0039)
[Sun Jul  9 09:56:24 2023] sd 6:0:39:0: device_unblock and setting to running, handle(0x003a)
[Sun Jul  9 09:56:24 2023] sd 6:0:40:0: device_unblock and setting to running, handle(0x003b)
[Sun Jul  9 09:56:24 2023] sd 6:0:41:0: device_unblock and setting to running, handle(0x003c)
[Sun Jul  9 09:56:24 2023] sd 6:0:42:0: device_unblock and setting to running, handle(0x003d)
[Sun Jul  9 09:56:24 2023] sd 6:0:43:0: device_unblock and setting to running, handle(0x003e)
[Sun Jul  9 09:56:24 2023] sd 6:0:44:0: device_unblock and setting to running, handle(0x003f)
[Sun Jul  9 09:56:24 2023] sd 6:0:45:0: device_unblock and setting to running, handle(0x0040)
[Sun Jul  9 09:56:24 2023] sd 6:0:46:0: device_unblock and setting to running, handle(0x0041)
[Sun Jul  9 09:56:24 2023] sd 6:0:47:0: device_unblock and setting to running, handle(0x0042)
[Sun Jul  9 09:56:24 2023] sd 6:0:48:0: device_unblock and setting to running, handle(0x0043)
[Sun Jul  9 09:56:24 2023] sd 6:0:49:0: device_unblock and setting to running, handle(0x0044)
[Sun Jul  9 09:56:24 2023] sd 6:0:50:0: device_unblock and setting to running, handle(0x0045)
[Sun Jul  9 09:56:24 2023] sd 6:0:51:0: device_unblock and setting to running, handle(0x0046)
[Sun Jul  9 09:56:24 2023] sd 6:0:52:0: device_unblock and setting to running, handle(0x0047)
[Sun Jul  9 09:56:24 2023] sd 6:0:53:0: device_unblock and setting to running, handle(0x0048)
[Sun Jul  9 09:56:24 2023] sd 6:0:54:0: device_unblock and setting to running, handle(0x0049)
[Sun Jul  9 09:56:24 2023] sd 6:0:55:0: device_unblock and setting to running, handle(0x004a)
[Sun Jul  9 09:56:24 2023] sd 6:0:56:0: device_unblock and setting to running, handle(0x004b)
[Sun Jul  9 09:56:24 2023] sd 6:0:57:0: device_unblock and setting to running, handle(0x004c)
[Sun Jul  9 09:56:24 2023] sd 6:0:58:0: device_unblock and setting to running, handle(0x004d)
[Sun Jul  9 09:56:24 2023] sd 6:0:59:0: device_unblock and setting to running, handle(0x004e)
[Sun Jul  9 09:56:24 2023] sd 6:0:60:0: device_unblock and setting to running, handle(0x004f)
[Sun Jul  9 09:56:24 2023] sd 6:0:61:0: device_unblock and setting to running, handle(0x0050)
[Sun Jul  9 09:56:24 2023] sd 6:0:62:0: device_unblock and setting to running, handle(0x0051)
[Sun Jul  9 09:56:24 2023] sd 6:0:63:0: device_unblock and setting to running, handle(0x0052)
[Sun Jul  9 09:56:24 2023] sd 6:0:64:0: device_unblock and setting to running, handle(0x0053)
[Sun Jul  9 09:56:24 2023] sd 6:0:65:0: device_unblock and setting to running, handle(0x0054)
[Sun Jul  9 09:56:24 2023] sd 6:0:66:0: device_unblock and setting to running, handle(0x0055)
[Sun Jul  9 09:56:24 2023] sd 6:0:67:0: device_unblock and setting to running, handle(0x0056)
[Sun Jul  9 09:56:24 2023] sd 6:0:68:0: device_unblock and setting to running, handle(0x0057)
[Sun Jul  9 09:56:24 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/7c80113e-a5b5-4290-b303-f6f2b6f7ce7f error=5 type=2 offset=14729805824 size=4096 flags=180880
[Sun Jul  9 09:56:24 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/5341cce4-6f16-461b-a952-9e5737430037 error=5 type=2 offset=11048407040 size=4096 flags=180880
[Sun Jul  9 09:56:24 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/7c80113e-a5b5-4290-b303-f6f2b6f7ce7f error=5 type=1 offset=270336 size=8192 flags=b08c1
[Sun Jul  9 09:56:24 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/7bd6210b-b670-4520-aadc-ea5923825993 error=5 type=2 offset=8594128896 size=4096 flags=180880
[Sun Jul  9 09:56:24 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/5341cce4-6f16-461b-a952-9e5737430037 error=5 type=1 offset=270336 size=8192 flags=b08c1
[Sun Jul  9 09:56:24 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/5341cce4-6f16-461b-a952-9e5737430037 error=5 type=1 offset=4000786423808 size=8192 flags=b08c1
[Sun Jul  9 09:56:24 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/5341cce4-6f16-461b-a952-9e5737430037 error=5 type=1 offset=4000786685952 size=8192 flags=b08c1
[Sun Jul  9 09:56:24 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/7c80113e-a5b5-4290-b303-f6f2b6f7ce7f error=5 type=1 offset=4000786423808 size=8192 flags=b08c1
[Sun Jul  9 09:56:24 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/7bd6210b-b670-4520-aadc-ea5923825993 error=5 type=2 offset=12275531776 size=4096 flags=180880
[Sun Jul  9 09:56:24 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/7bd6210b-b670-4520-aadc-ea5923825993 error=5 type=2 offset=9821261824 size=4096 flags=180880
[Sun Jul  9 09:56:24 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/7bd6210b-b670-4520-aadc-ea5923825993 error=5 type=1 offset=270336 size=8192 flags=b08c1
[Sun Jul  9 09:56:24 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/7bd6210b-b670-4520-aadc-ea5923825993 error=5 type=1 offset=4000786423808 size=8192 flags=b08c1
[Sun Jul  9 09:56:24 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/7bd6210b-b670-4520-aadc-ea5923825993 error=5 type=1 offset=4000786685952 size=8192 flags=b08c1
[Sun Jul  9 09:56:24 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/7c80113e-a5b5-4290-b303-f6f2b6f7ce7f error=5 type=1 offset=4000786685952 size=8192 flags=b08c1
[Sun Jul  9 09:56:24 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/a20862c1-7f7e-4dc0-b261-8485793bb170 error=5 type=2 offset=8594128896 size=4096 flags=180880
[Sun Jul  9 09:56:24 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/a997b627-aa70-4c51-ae25-b9f8adf519f8 error=5 type=2 offset=8594128896 size=4096 flags=180880
[Sun Jul  9 09:56:24 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/a20862c1-7f7e-4dc0-b261-8485793bb170 error=5 type=1 offset=270336 size=8192 flags=b08c1
[Sun Jul  9 09:56:24 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/a997b627-aa70-4c51-ae25-b9f8adf519f8 error=5 type=2 offset=12275531776 size=4096 flags=180880
[Sun Jul  9 09:56:24 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/a997b627-aa70-4c51-ae25-b9f8adf519f8 error=5 type=1 offset=270336 size=8192 flags=b08c1
[Sun Jul  9 09:56:24 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/a997b627-aa70-4c51-ae25-b9f8adf519f8 error=5 type=1 offset=4000786423808 size=8192 flags=b08c1
[Sun Jul  9 09:56:24 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/a997b627-aa70-4c51-ae25-b9f8adf519f8 error=5 type=1 offset=4000786685952 size=8192 flags=b08c1
[Sun Jul  9 09:56:24 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/a20862c1-7f7e-4dc0-b261-8485793bb170 error=5 type=1 offset=4000786423808 size=8192 flags=b08c1
[Sun Jul  9 09:56:24 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/6fc6792d-f095-4768-865a-1a5be5e9c636 error=5 type=2 offset=9821261824 size=4096 flags=180880
[Sun Jul  9 09:56:24 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/a20862c1-7f7e-4dc0-b261-8485793bb170 error=5 type=1 offset=4000786685952 size=8192 flags=b08c1
[Sun Jul  9 09:56:24 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/6fc6792d-f095-4768-865a-1a5be5e9c636 error=5 type=2 offset=11048407040 size=4096 flags=180880
[Sun Jul  9 09:56:24 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/6fc6792d-f095-4768-865a-1a5be5e9c636 error=5 type=1 offset=270336 size=8192 flags=b08c1
[Sun Jul  9 09:56:24 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/6fc6792d-f095-4768-865a-1a5be5e9c636 error=5 type=1 offset=4000786423808 size=8192 flags=b08c1
[Sun Jul  9 09:56:24 2023] zio pool=hddtest vdev=/dev/disk/by-partuuid/6fc6792d-f095-4768-865a-1a5be5e9c636 error=5 type=1 offset=4000786685952 size=8192 flags=b08c1
[Sun Jul  9 09:56:24 2023] WARNING: Pool 'hddtest' has encountered an uncorrectable I/O failure and has been suspended.

After all these, the SAS_HDD pool is suspended but the SAS_SSD pool is working without any problem.

The SAS_HDD's are "SEAGATE ST4000NM017A" and they are brand new. I checked all of the drives with smartctl and all passed the long-self-test. 

The problem only occurs at boot sequence, if I disable zpool auto-import, I can manually import the pool and use without any issue. 

mpt3sas_cm0: LSISAS3316: FWVersion(13.00.08.00), ChipRevision(0x01), BiosVersion(15.00.06.00)

I checked a new firmware version for UCS-S3260-DHBA because 13.00.x.x pretty old but couldn't find a new firmware. 

Can you help me please? 

[marce1000]

      - You may want to post that in           https://www.truenas.com/community/forums/hardware/

 M.