Client roaming between APs at high traffic load cause 3 second gap

[Anders Hedlund]

We are operating our vehicles remotely in the underground mines over Wi-Fi, the data load from the machine is constant around 30Mbit/s (yes, six to eight HD-cameras).
When switching between APs we have noticed that we sometimes get a transmission gap of around 3 seconds, causing the safety mechanism on the trucks to stop the machine.
We have activated Radio Active Traces in the WLC and gets the response included below.
We have more than 150 access points, and this happens between a few of them, most of them never has this problem, but on a few one it happens, we have tried changed the cables and also the access points (we even switched model).
We have tested roaming between them with low load and this does not happen then.
We are on WLC 9800-L version 17.13.1 (but we had the issue on the 17.9.4A also).
The AP version/models are:
17.13.0.107, C9124AXE-E
17.13.0.107, C9124AXE-B
17.13.0.107, C9120AXE-E
This is the reported issue in the debugTrace:
2024/04/04 04:07:20.696836577 {wncd_x_R0-0}{1}: [client-orch-sm] [18090]: (info): MAC: 0090.e89c.47bd Deleting the client, reason: 286, CO_CLIENT_DELETE_REASON_MN_AP_DRIVER_EVENT_CLASS3_RECV, Client state S_CO_RUN
2024/04/04 04:07:20.697072661 {wncd_x_R0-0}{1}: [errmsg] [18090]: (debug): %CLIENT_ORCH_LOG-7-CLIENT_MOVED_TO_DELETE_STATE: R0/0: wncd: Username (null), MAC: 0090.e89c.47bd, IP 10.88.19.154 disconnected from AP (G1N-AP-S-356-1) with SSID (Voysys-AX)
2024/04/04 04:07:20.697082437 {wncd_x_R0-0}{1}: [client-orch-sm] [18090]: (note): MAC: 0090.e89c.47bd Client delete initiated. Reason: CO_CLIENT_DELETE_REASON_MN_AP_DRIVER_EVENT_CLASS3_RECV, details: , fsm-state transition 2c|2d|2e|15|1a|1b|2c|37|46|48|4a|4c|51|60|62|83|86|8e|15|1a|1b|2c|37|46|48|4a|4c|51|60|62|83|ab|

And in the analysis tool, we get:
Controller initiated client deletion with code: CO_CLIENT_DELETE_REASON_MN_AP_DRIVER_EVENT_CLASS3_RECV. Explanation: AP triggered delete, as client sent class 3 frame (data, power save, some management) from non-authenticated client, may happen occasionally on connection recovery. Actions: Nothing required, unless this is happening frequently. This may need RA trace and OTA capture
Has anyone seen this? Any ideas how we can debug this further to better find a solution or work-around?
/Anders

[marce1000]

    You can feed Radio Active traces  into : Wireless Debug Analyzer for high level analysis
    Outputs from https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-wireless-controllers/217738-monitor-catalyst-9800-kpis-key-performa.html#anc5
     can also be useful

   - You may try to tweak e.g. parameters related to fast roaming (802.11k,...)
   -  I can assume that the mining environment is very challenging for wireless ; has sufficient site survey been done ?
   - Check if wireless drivers of the camera's are up to date , and or can be updated or else  , look at new revisions of the
   camera models with other firmware if applicable , 

  Also have a checkup of the WLC 9800-L configuration with the CLI command show tech wireless and feed the output to : Wireless Config Analyzer

  M.

[Anders Hedlund]

Thanks Marce, yes, we have gone through the usual troubleshooting processes, what makes it tricky is that it's so tight correlated to the traffic load. We have tested adding a UDP packet generator (then skipping the cameras, so only the safety mechanism and the UDP packets) and if we set it to like 30Mbit/s UDP packets can clearly see the problem, but not if we switch it off or have less load).

[Leo Laohoo]

CSCwi49862

[marce1000]

 - @Leo Laohoo Leo , do you have shared and or allowed content availability for this bug because I get :
                                 This bug is not available for viewing at this time

 M.

[Leo Laohoo]

https://imgur.com/pKY4D6t

[Anders Hedlund]

Now this was interesting, thanks, we might have some contact that can dig this one out for us (it has some security classification).

[JPavonM]

@Anders Hedlund have you tested 17.12 train?

Cisco only recommends 17.13 for deployments that need new features introduced in that code, but TC will tell you to move to recommended 17.9.5 or 17.12.2 (17.12.3 in few weeks)

[Leo Laohoo]

17.12.3 has been out since 21 March 2024 and 17.14.1 was released last week, 13 April 2024.

Release Notes for Cisco Catalyst 9800 Series Wireless Controller, Cisco IOS XE Dublin 17.12.x

Release Notes for Cisco Catalyst 9800 Series Wireless Controller, Cisco IOS XE 17.14.x

[Anders Hedlund]

Yes, we have to use 17.13, we have many many AP model 9124, and we need 17.13 to be able to run then in indoor-mode enabling non-DFS channel usage (we can not use DFS channels in our setup, it causes too long communucation gaps so the machines will make a safety stop).

[Leo Laohoo]

@Anders Hedlund,

Since the APs are in a vehicle, are the APs powered OFF when the vehicle is not in use?

[Anders Hedlund]

We don't have the APs on the vehicles, sorry for not beeing clear about this. On the vehicles we have MOXA 4131A clients.

[Leo Laohoo]

---

@Anders Hedlund wrote:
We don't have the APs on the vehicles, sorry for not beeing clear about this. On the vehicles we have MOXA 4131A clients.

---

Good. 

Reboot the APs and see if the roaming has improved.  If it has, consider the possibility of rebooting the APs daily -- Read between the lines.