AP image predowload process prior to the upgrade of a WLC

[Aleck_Sei]

Hello everyone,

Soon we will have to perform the firmware update to one of our 8540 WLC It is currently running with version 8.5.161.0 and has about 5500 APs registered. On previous occasions, we have managed to do it although we have observed behavior that we do not fully understand.

To minimize the network impact time, once the new image is downloaded to the controller we use the 'pre-download' procedure for the APs to download that image:

config ap image predownload primary all

We use the 'all' option to run it on all of them but always, around 35-40% remain in the 'Failed' state and do not perform the pre-download. We have verified that what happens with any model of the ranges that we use (17xx, 37xx, 28xx, 38xx,...), some of them carry out the process correctly and others of the same model do not.
We know that there is a limitation in the WLC that limits the process to 1000 simultaneous downloads. Our theory is that the rest of the downloads should be done at as those in progress end but that is not the case. There is a moment when everyone starts to fail.

Does anyone know what this could be due to? Are we making a mistake? Could it be due to some kind of overloading of the WLC by asking it to download so many APs? You can pre-download 'bit little by little' launching by groups of AP, for example, 500 at a time?

Thank you in advance for your help.

[marce1000]

                                                            >...Does anyone know what this could be due to?
                              - Configure a syslog server on the controller and for all APs too , to get insights :
              https://www.cisco.com/c/en/us/support/docs/wireless/4100-series-wireless-lan-controllers/107252-WLC-Syslog-Server.html
   especially  : https://www.cisco.com/c/en/us/support/docs/wireless/4100-series-wireless-lan-controllers/107252-WLC-Syslog-Server.html#toc-hId-208802317

       Bugs like this one could be involved : https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvg41698
                                                                https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvr41180

 M.
         

[Leo Laohoo]

What firmware version is the controller planned to upgrade to? 

Give me the exact numbers of the AP base on their model numbers.  I need this info because I have a "hack" that will force all the APs to "pseudo" pre-download at the same time (bypassing the 1000 AP limit).  

Before I forget:

1.  The 2800/3800 belong to the same "family".  This also means one IOS file is only needed, meaning, the IOS file for the 2800 and the IOS file for the 3800 are identical.  

2.  APs running on Cheetah OS (example, 2800/3800) are easy to upgrade because it takes <60 seconds for an AP to download the firmware and another 20 seconds to load into the flash.  

[Aleck_Sei]

Thanks for your help!

We plan to update the WLC to version 8.10.183.0

These are the APs that are currently hanging from it:

AIR-AP1562E-E-K9 --> 4
AIR-AP2802I-E-K9 --> 2000
AIR-AP3802I-E-K9 --> 3200
AIR-CAP1532E-E-K9 --> 120
AIR-CAP1602I-E-K9 --> 80
AIR-CAP1702I-E-K9  --> 80
AIR-CAP3702I-E-K9 --> 60

[Rich R]

1. 1602 is not supported on 8.10 software.  The AP is End of Support and was last supported on 8.5.  Refer to the Compatibility Matrix link below.  So that's 80 APs you will need to upgrade.

2. If you're going to the trouble of updating the software then it should be to 8.10.190.0 not 8.10.183.0.  Refer to the TAC recommended link below.

[Leo Laohoo · Accepted solution]

---

@Aleck_Sei wrote:

Thanks for your help!

We plan to update the WLC to version 8.10.183.0

These are the APs that are currently hanging from it:

AIR-AP1562E-E-K9 --> 4
AIR-AP2802I-E-K9 --> 2000
AIR-AP3802I-E-K9 --> 3200
AIR-CAP1532E-E-K9 --> 120
AIR-CAP1602I-E-K9 --> 80
AIR-CAP1702I-E-K9  --> 80
AIR-CAP3702I-E-K9 --> 60

---

Right, this is going to be very easy.  

Due to the download limitation imposed on the OS (1000 APs downloading), I want to concentrate on the APs running Cheetah OS, namely the 1560, 2800 and 3800.  The rest of the APs will just download normally (less than 1000 APs anyway). 

The WLC will upgrade to 8.10.183, right?  So here is how it goes:  

1.  If I look under Supported Access Points in Cisco AireOS Wireless Software Releases correlates to 15.3(3)JK8b. 
2.  Download 15.3(3)JK8b for the 2800/3800/1560 (filename:  ap3g3-k9w8-tar.153-3.JK8b.tar) and put the file in a TFTP that is accessible from the APs in question. 
3.  Do the following from the WLC:  

debug ap enable <AP NAME>
debug ap command "archive download-sw /no-reboot tftp://<IP ADDRESS>/ap3g3-k9w8-tar.153-3.JK8b.tar" <AP NAME>

4.  Depending on the bandwidth of the TFTP server to the AP, wait for about 3 minutes.  Once the AP has finished downloading the firmware, forcefully reboot the CoS AP: 

config ap reset <AP NAME>

5.  Within 4 minutes after forcefully rebooting the AP, reboot the WLC.  

Let me explain what the steps are: 

Step 3 is an instruction for the WLC to "instruct" the AP to download the firmware.  In Step 4, the APs must be rebooted to load the new 8.10.183.0 firmware.  While the APs reboot (Step 5), the WLC reboots into 8.10.183.0.  

When the WLC comes online the CoS APs, already booted up and loaded with 8.10.183.0, will join the WLC automatically.  The classic IOS, numbering <1000, will download the firmware from the controller and join 20 minutes later.  

With 5204 CoS, I use an Excel spreadsheet to generate a formula.  I cut-n-paste the commands into the WLC about 50 lines at a time and do not stop until I finish the entire list.  

[Rich R]

Yes capacity could be an issue and you might need to try repeated pre-downloads to get most downloaded.  There were also some bugs in older code with the download numbers getting out of sync which might be a factor.

The other thing to watch out for on the 17xx and 37xx is flash corruption.
Check https://www.cisco.com/c/en/us/support/docs/field-notices/703/fn70330.html and https://www.cisco.com/c/en/us/support/docs/wireless-mobility/wireless-lan-wlan/213317-understanding-various-ap-ios-flash-corru.html for more details.

If your APs are at remote sites (configured for flexconnect) then you might want to consider the FlexConnect AP Image Upgrade feature. https://www.cisco.com/c/en/us/td/docs/wireless/controller/8-10/config-guide/b_cg810/configuring_flexconnect_ap_upgrades_for_flexconnect_aps.html
This uses CAPWAP to download the image to one of each type of AP at the site and then the other APs TFTP the image from the primary AP(s) locally.  The can improve pre-download time and reliability if remote sites have limited bandwidth and long round trip delay.

[Aleck_Sei]

Thanks for your help @Rich R !

The idea of ​​using the 'FlexConnect AP Image Upgrade' functionality seems great because we have it activated in our environment!

If I have not misunderstood the procedure, would you try to do the predownload for each of the FlexConnect group individually?

config flexconnect group <GROUP_1> predownload start primary
config flexconnect group <GROUP_2> predownload start primary
config flexconnect group <GROUP_3> predownload start primary
....
config flexconnect group <GROUP_N> predownload start primary

[Rich R · Accepted solution]

Yes I believe that's the way it works @Aleck_Sei 
Obviously it means having to set up flexconnect groups if you haven't already.
On 9800 it's a more general feature which is enabled by default.

[Aleck_Sei]

Thank you very much for your help @Rich R