9166 clients not connecting at 5 GHz

[eglinsky2012]

9800-80 pair running 17.9.4a and APSP8 (upgraded about a month ago). Upon investigating after slowness/connection complaints from the professor of a 150-seat lecture hall where 9166s are installed, I realized that every single client in the room was connected at 2.4 GHz. I realized that RRM had set the 2.4 GHz radios higher than the 5 GHz ones, which I corrected, but clients still wouldn't connect at 5 GHz. I visited the room and my iPhone 12 Mini and Dell laptop with Intel AX211 both connected at 2.4 GHz, even though I sat right under an AP.

We have an identical lecture hall next door, and last week I saw that clients were connecting at 5 GHz, but now all are connecting at 2.4 GHz. It seems like this could become a bigger issue.

I took an OTA, packet capture at the WLC, RA trace, and tech support from the AP before and after rebooting the APs. Upon rebooting, all the clients are connecting at 5 GHz. Spectrum analysis looked clean. I opened a TAC case yesterday and they are analyzing the information collected, but I thought I'd post to see if anyone else was experiencing this.

Here's the client distribution in the two rooms. The ones named BPB-130 are the ones that were rebooted, the ones in BPB-131 have not been rebooted yet. Notice the opposite trend of 2.4 vs 5 GHz client counts.

[Haydn Andrews]

I have found with Apple devices once they roam to 2.4 they very rarely move back to 5GHz. Basically the current signal needs to be above -70 and the 5GHz has to be -65

But given you seen Dell doing same probably rules that out

SSID have band select enabled? Did you try bouncing just one AP first and see if it got clients on 5GHz?

Would like to hear the TAC response on this one
Also I generally set the 2.4 TX limits in the RF profile to lower than the 5GHz ones

[eglinsky2012]

Right, I don't think this is device specific since it's a 150-seat lecture hall at a university, so it's BYOD, which means a mix of iOS, macOS, Windows, and Android (and whatever else).

I have band select disabled. It doesn't seem to be necessary anymore with the vast majority of devices not only supporting but preferring 5 GHz on their own (as the client counts on the APs named BPB-130 in the screenshot show; this is typical).

I did not try one AP at a time. I'll suggest that next time I discuss with TAC. For now, I'm waiting to hear back if they want to do more troubleshooting before I reboot the APs in the other room (BPB-131). While I understand the importance of collecting data while the issue is occurring, I can't leave it this way too long.

I forgot to mention that I did try bouncing the 2.4 GHz radios and none of the clients moved to 5 GHz. So I bounced the 5 GHz radios and then the 2.4 GHz again, and they still didn't move. Only rebooting the APs in the one room restored functionality (for now).

Yeah, I have work to do on our default RF profile. For now, I have all the channels and power levels set manually per our standard practice for lecture halls. Around 14 dBm for 5 G Hz and 8 dBm for 2.4 GHz. I have them set this way in both rooms.

[marce1000]

 - Have a checkup of the primary controller configuration using the CLI command show tech wireless and feed the output into :
                                                                                                                         Wireless Config Analyzer

 M.

[eglinsky2012]

Already done, that did not show anything relevant.

@JPavonM I had tried disabling 2.4 momentarily, but I didn’t leave it off for a prolonged time. I might give that a try as well.

[JPavonM]

Try disabling it before students enter the classroom so no device would connect to 2.4 before.

In parallel, this way you will know if all of them are 5-GHz capable so you can leave 2.4 radio disabled (if desired).

[eglinsky2012]

Both rooms have classes starting shortly, so students are filtering in.

Current state is same as before; all clients on 5 G Hz in room 130 but all on 2.4 in room 131:

After disabling 2.4 in room 131 for a few minutes, the client count in room 130 is increasing, but the client count in room 131 is zero. In fact, there are now a bunch of 2.4 GHz clients in room 130 -- I'm guessing those are clients in room 131 that are forced to connect to 130 because I have the 2.4 GHz radios disabled in room 131 and the 5 GHz radios are not functioning there.

A few minutes later, just before start of class:

I re-enabled 2.4 GHz in room 131 at the start of class and, after waiting 5 minutes, the clients have reconnected to the APs in that room, yet again all at 2.4 GHz, while only a few clients in room 130 are left on 2.4 GHz.

[marce1000]

  - Also check . https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-wireless-controllers/217738-monitor-catalyst-9800-kpis-key-performa.html#anc6
                  https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-wireless-controllers/217738-monitor-catalyst-9800-kpis-key-performa.html#anc5

 M.
   

[JPavonM]

Have you tested with disabling 2.4 GHz to see how does it like?

[eglinsky2012]

Over time, I have found a few more rooms affected by this issue, including one that has 9130E access points rather than 9166I. The 9130Es are on the same 9800 controller and 17.9.4a/APSP8 as the 9166s. TAC case has been open since 2/8, nothing from them yet. Meanwhile, I have Prime rebooting the APs in the affected rooms every M, W, F and that has kept them functional.

[Leo Laohoo]

---

@eglinsky2012 wrote:
Only rebooting the APs in the one room restored functionality (for now).

---

Really sounds like a software bug, alright.

Remove APSP8 from the selected APs and see if it improve things.  Alternatively, how about upgrading to 17.9.5?

[eglinsky2012]

For now, I’m maintaining status quo until TAC tells me otherwise. 17.9.5 would have been a nice idea for spring break in a couple weeks, but we are migrating uplinks off 6500s to 9500s and our guest captive portal off firewalls to ISE, so I don’t want to muddy the waters with a controller code upgrade in case something (else) goes wrong as a result of that work.

[eglinsky2012]

TAC suggested disabling 802.11r due to the following logs collected: CO_CLIENT_DELETE_REASON_FT_AUTH_RESPONSE

I put a hold on that because of issues that occurred after the aforementioned migration.

Meanwhile, thanks to @Leo Laohoo for posting this bug in another thread: CSCwj45141 Sounds like the issue at hand. I shared the bug with TAC to see if they have any insight on planned fixes. For now, the Mon-Wed-Fri scheduled reboots a la Prime have maintained functionality of the 5 GHz radios.

[JPavonM]

If you know all client devices do support 5GHz band, why keeping the SSID using 2.4 GHz band?

On these scenarios is best to use 5 GHz only, and create a separate SSID with "-legacy" suffix only for 2.4 GHz for those legacy devices that do not support 5 GHz.