WLC 9800 CPU Utilize Issue

[bakaholic39]

Hi,

I have a WLC 9800 on Firmware OS : 17.9.3
19 APs and 100 Clients 

Everything looks normal, but there are reports that the CPU utilization rate is too high. The problem is only 7th core. Has anyone encountered the same problem as me. What should I do to solve it?

Thank you!

[Leo Laohoo]

Please share the complete output to the following commands: 

sh platform resources
sh platform software status con brief

[bakaholic39]

Here are the results.

 

 Thank you for your response.

[Leo Laohoo]

NOTE:  The two commands are a "real-time snapshot".   It is not historical. 

The first one looks at the data plane.  The value are normal. 

The second one looks at the control plane.  And the memory utilization of 30% is not normal.   

One of the CPU, #7, is grinding.  

Please post the complete output to the next command:  sh processes cpu platform sorted location 0

NOTE:  This output will produce a very long "page".  I just want to see the "first" page.

[bakaholic39]

With this command, all process seems very normal except #7 CPU


After that I tried this command. There seems to be something wrong with process " ucode_pkt_PPE0 "
My question is why doesn't the system share load with other cores.


Thank you for your response.

[Leo Laohoo]

Raise a TAC Case with EUMA or NAM/LATAM.  I am out of answers to what this process does.  

[Rich R]

High ucode_pkt_PPE0 is "normal" - I seem to recall there's a recent change to how it's displayed (I think CSCve73211 mentioned below) but you can see mention of it here https://blogs.cisco.com/networking/wireless-catalyst-9800-wlc-kpis-part-1

As to the high CPU on that core - look at your wncd's: show process cpu platform sorted | incl wncd

If you confirm wncd causing it then look at why. 
19 APs and 100 clients is very small so that's surprising but check through best practise guide on AP tagging for distribution across wncd's.
Also, if you're using web-auth - turn OFF https redirects - they are very CPU hungry and will cause high CPU and client issues.

[jagan.chowdam]

Check CPU utilization: Identify if there is any process using lot of CPU using the following command
sh processes cpu platform sorted | ex 0% 0% 0%

Check CPU utilization per wncd process
show processes cpu platform sorted | i wncd

Check Memory utilization: Identify if device is running out of memory and if there is some process holding lot of memory.
show platform resources (basic)
show process memory platform sorted
show processes memory platform accounting (advanced)

CJ

/**Please rate all useful responses**/

[bakaholic39]

Here are the results

 

Is there any problem with process ucode_pkt_PPE0?

Thank you for your response.

[jagan.chowdam]

For CPU monitoring we need to run following command multiple times. See if there are any processes showing CPU above 80% consistently and not just a spike.

sh processes cpu platform sorted | ex 0% 0% 0%

Catalyst 9800-CL and 9800-L platforms use CPU cores for data forwarding. Therefore, it is expected to see high CPU in ucode_pkt_PPE0. For those platforms to evaluate data plane performance use command: 

show platform hardware chassis active qfp datapath utilization | i Load

Refer the following Cisco 9800 KPI blog for detailed explanation as which commands to run.

https://blogs.cisco.com/networking/wireless-catalyst-9800-wlc-kpis-part-1

CJ

[calverthall]

Hello,  Did you get any clarity on this issue?  We just upgraded to 17.9.3 and noticed the exact same thing.  I just opened a TAC, but figured I would ask.

Thanks, Austin

[bakaholic39]

Still no solution. I already opened TAC but he replied me that it's normal state without giving any further information.

[calverthall]

Thanks.  I received the "it is expected to see high CPU in  one of the cores, basically this is a normal behavior and it could be on any core, you always will see one of the cores on high utilization" response as well.

I am not 100% convinced I would be getting alerts like this during the summer (we are a school that is out of session) and that this was normal prior to the upgrade.  My other thought would be why have 10 cores if only one runs at 99.7%...

Austin

[calverthall]

Here is the full response I received.  Still not sure why an interface has a RED Alert indicating high utilization, but I asked them to share that with the UI team.  Hopefully we can blissfully ignore...  - Austin

The global figures reflect the actual overall resources, in this case you are in green numbers.

The readings are normal across the board and Cisco documentation indicates that it is expected that IOS-XE platforms show high CPU in the ucode_pkt_PPE0 process even when sitting idle.

Please refer to this article:  https://bst.cisco.com/bugsearch/bug/CSCve73211

To improve packet throughput and reduce packet latency and jitter, the data plane more aggressively polls for new packets in later releases which raises CPU utilization even at low packet rates.

[bakaholic39]

Here is the same response from Cisco wireless team

Please note that the high CPU 7 Utilization in "ucode_pkt_PPE0" for 9800 IOS-XE 16.4 and higher is expected behavior to improve packet throughput and reduce packet latency and jitter, this increases the number of CPU cycles that "ucode_pkt_PPE0" process consumes at baseline even without any traffic.

The "ucode_pkt_PPE0" is often at 100% even if the box is not passing that much traffic. It is normal for that process to report high usage, it's not an appropriate metric of how much traffic is passing through.

Reference: https://bst.cloudapps.cisco.com/bugsearch/bug/CSCve73211