3802I 启动死循环

[aironetlap]

本地是一个由 4 台 3802I-E-K9 组成的 Mobility Express 环境，位于同一个 L2 下。其中三台已经接入了 ME 虚拟控制器管理。

剩下的一台一直困在part1 镜像损坏 的问题中，并不断重启。

Starting kernel ...

[01/01/1970 00:00:00.0000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 260096
[01/01/1970 00:00:00.0000] Memory: 1025736K/1048576K available (5743K kernel code, 411K rwdata, 2504K rodata, 359K init, 496K bss, 22840K reserved, 0K highmem)
[01/01/1970 00:00:00.1200] CPU1: Booted secondary processor
[01/01/1970 00:00:01.4600] buginf tty flushing thread started, ttyport=be8c1c00
[01/01/1970 00:00:01.5500] m25p80 spi1.0: found mx25l3206, expected n25q032
[*01/01/1970 00:00:02.7183] buginf() enabled.
[*01/01/1970 00:00:02.7280] Made it into bootsh: Sep 29 2023 03:17:16 T-eea26d9aecddbce8a3d538e497d7baad0ab582d0-geea26d9a-aut
[*01/01/1970 00:00:04.1770] verify signature failed for /bootpart/part1/ramfs_data_cisco.cpio.lzma
[*01/01/1970 00:00:04.1771] bootsh mini ramfs booted /bootpart/part1/ramfs_data_cisco.cpio.lzma
[*01/01/1970 00:00:22.3785] lzma: unexpected EOF
[*01/01/1970 00:00:22.3805] Uncompressing lzma file: /bootpart/part1/ramfs_data_cisco.cpio.lzma: File exists
[*01/01/1970 00:00:22.3806] Fatal error: failed to start the image. Please fall back to alternate partition...
[01/01/1970 00:00:22.6800] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100
[01/01/1970 00:00:22.6800] 
[01/01/1970 00:00:22.6800] CPU1: stopping
[01/01/1970 00:00:22.6800] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 3.14.33 #1
[01/01/1970 00:00:22.6800] Backtrace: 
[01/01/1970 00:00:22.6800] [<80109814>] (dump_backtrace) from [<80109ae0>] (show_stack+0x18/0x1c)
[01/01/1970 00:00:22.6800]  r6:00000000 r5:809cf174 r4:00000000 r3:00200040
[01/01/1970 00:00:22.6800] [<80109ac8>] (show_stack) from [<8068d494>] (dump_stack+0x70/0x8c)
[01/01/1970 00:00:22.6800] [<8068d424>] (dump_stack) from [<8010c8f0>] (handle_IPI+0xe0/0x238)
[01/01/1970 00:00:22.6800]  r4:00000001 r3:80979a58
[01/01/1970 00:00:22.6800] [<8010c810>] (handle_IPI) from [<801004d4>] (gic_handle_irq+0x58/0x60)
[01/01/1970 00:00:22.6800]  r7:bf0adfac r6:80971434 r5:bf0adf78 r4:00000005
[01/01/1970 00:00:22.6800] [<8010047c>] (gic_handle_irq) from [<806944c0>] (__irq_svc+0x40/0x50)
[01/01/1970 00:00:22.6800] Exception stack(0xbf0adf78 to 0xbf0adfc0)
[01/01/1970 00:00:22.6800] df60:                                                       bf7e66e0 00000000
[01/01/1970 00:00:22.6800] df80: 0000a104 801159e0 bf0ac000 bf0ac000 10c03c7d 809cf194 0000406a 414fc091
[01/01/1970 00:00:22.6800] SMP: failed to stop secondary CPUs
[01/01/1970 00:00:22.6800] Rebooting in 5 seconds..
[01/01/1970 00:00:22.6800] dfa0: 00000000 bf0adfcc bf0adfd0 bf0adfc0 80106ef8 80106efc 60000013 ffffffff
[01/01/1970 00:00:22.6800]  r6:ffffffff r5:60000013 r4:80106efc r3:80106ef8
[01/01/1970 00:00:22.6800] [<80106ec8>] (arch_cpu_idle) from [<80160950>] (cpu_startup_entry+0x138/0x1cc)
[01/01/1970 00:00:22.6800] [<80160818>] (cpu_startup_entry) from [<8010c594>] (secondary_start_kernel+0x124/0x144)
[01/01/1970 00:00:22.6800] [<8010c470>] (secondary_start_kernel) from [<00100564>] (0x100564)
[01/01/1970 00:00:22.6800]  r4:3f09406a r3:8010054c

细节描述如下：

1. part1 为最新版本镜像，part2 为旧版本 CAPWAP 镜像

2. 在 u-boot 手动设定 `setenv BOOT part2` 或者连续 5 次启动 part1 失败后，设备会从 part2 启动。但启动后，设备会从 ME 控制器中获取对应的 ME CAPWAP 镜像，写入到 part1，然后收到控制器的 reset request，从 part1 启动，再次陷入循环。

[*04/17/2024 16:02:58.0569] Sending Join request to 192.168.0.16 through port 5272
[*04/17/2024 16:02:58.0601] Join Response from 192.168.0.16 
[*04/17/2024 16:02:58.0601] TLV-DEC-PROC: Info missing for 1363
[*04/17/2024 16:02:58.0601] TLV-DEC-PROC: Info missing for 786
[*04/17/2024 16:02:58.1286] HW CAPWAP tunnel is ADDED
[*04/17/2024 16:02:58.1409] CAPWAP State: Image Data
[*04/17/2024 16:02:58.1715] do ACTIVATE, part2 is active part
[*04/17/2024 16:02:58.2669] activate part1, set BOOT to part1
[*04/17/2024 16:02:58.9105] AP primary version: 8.10.190.0
[*04/17/2024 16:02:58.9145] AP backup version: 8.2.166.0
[*04/17/2024 16:03:01.9191] AP Rebooting: Reset Request from Controller(Image Upgrade)

已经尝试过以下操作

1. 启动时长按 mode 按钮，直至 console 输出中计时器超过 20 秒，恢复出厂设置。

2. 在 u-boot 环境下使用上文链接中的命令擦除 part1 分区。此后 ap 会因为 part1 为空，fallback 到 part2 启动，然后从 ME 拉取新镜像至 part1，再次进入启动循环。

我没有服务合同，所以无法开 TAC ticket。感谢各位的帮助！

[aironetlap · Accepted solution]

问题已解决。

part2 运行的是 8.2.166.0 古老版本，但 ME 运行 8.10.190.0 最新版本。子节点必须从 part1 启动，而从低版本直接升级到 8.10.190.0 版本会导致 broken image，原因不明。

要摆脱这个 deadlock 情况，需要将故障 ap 隔离出来单独从健康的 part2 启动，然后使用 `archive download-sw` （写入 image 到另一个分区）和 `config boot path`（切换启动分区）逐步将两个分区升级到中间版本（如 8.5.160.0），再升级到 8.10.190.0。最后，在启动时长按 mode 直到 console timer 到达 20s 后，将此节点配置抹除（因为创建了临时的 ME 控制器配置），以干净的状态加入到现有的 ME cluster 下。这样问题就解决了。