C9117AXI stuck in EWC-AP Recovery mode

[peat]

Hi,

I was given an 9117ax to setup with a SSID to use for a wireless survey.  But I think i got confused about what models were available and I thought i had a lightweight mode AP when in fact it was already the embedded WLC version.

The long story short is I followed the cisco guide on converting a lightweight AP to a standalone AP and even though everything seem to go fine after the tftp upload of the AP image and the WLC bin file on the reboot the AP has gone into EWC-AP recovery mode and keeps saying 

"EWC-AP Recovery Mode",  Run archive download-sw "

So I tried that archive download command and it looked like it went through fine till it got to 72.6% and then it said error .

I tried again and it stopped at 72.6% again.

I did a show flash and it showed  that there was only 187mb left but the wlc.bin is 280mb so im guessing thats my problem?

But after googling how to make space it doesnt look like a switches/routers output and the list of files in the flash doesnt have filesize or are recognisable to me.

I seen somewhere mention show flash cores but that said there was nothing there.

I've tried a factory reset but that didn't change anything.

Can anyone help get this AP back to working ?

The model is C9117AXI-E which after reading up is the EWC version so it looks like I really messed up and have tried to put the WLC image on a WAP that already had it on it.

[Scott Fella]

Did you use the ap1g6 image from the tar file you downloaded?  You can always try to delete some of the files on flash to free up some space is that is the reason.  Maybe post what you see so we have a better idea.

Here is a guide I used back in the day's for site survey.

Cisco 802.11ax site survey (single AP method) | SC-WiFi

[peat]

Hi,  

Yes used the ap1g6 image.

Yes I would like to delete some files from the flash.  Im sure that would fix the problem.

This is the list

AP1#show flash

Directory of /storage/
total 56K
-rw-r--r-- 1 root root 2 Dec 4 08:12 BOOT_COUNT
-rw-r--r-- 1 root root 2 Dec 4 08:12 BOOT_COUNT.reserve
-rw-r--r-- 1 root root 28 Dec 4 08:12 RELOADED_AT_UTC
drwxr-xr-x 3 rsyncuse root 224 Dec 4 08:12 application
-rw-r--r-- 1 root root 394 Dec 4 08:15 base_capwap_cfg_info
-rw-r--r-- 1 root root 2 Dec 4 07:19 boot_debug
-rw-r--r-- 1 root root 0 Dec 4 07:20 bootloader.log
-rw-r--r-- 1 root root 18 Dec 4 08:11 config
-rw-r--r-- 1 root root 0 Dec 4 08:12 config.local
-rw-r--r-- 1 root root 0 Dec 4 08:12 config.mobexp
-rw-r--r-- 1 root root 0 Dec 4 08:11 config.mobexp.tmp_5793
-rw-r--r-- 1 root root 0 Dec 4 08:12 config.oeap
-rw-r--r-- 1 root root 0 Dec 4 08:11 config.tls_client.tmp_5793
drwxr-xr-x 2 root root 160 Aug 20 2020 cores
drwxr-xr-x 2 root root 160 Jan 1 1970 dropbear
lrwxrwxrwx 1 root root 31 Dec 4 07:20 iot_status_iot0.txt -> /storage/iot_status_ttyMSM1.txt
-rw-r--r-- 1 root root 153 Dec 4 07:20 iot_status_ttyMSM1.txt
-rw-r--r-- 1 root root 213 Dec 4 07:49 last_good_uplink_config
drwxr-xr-x 2 root root 160 Aug 20 2020 lists
-rw-r--r-- 1 root root 63 Aug 20 2020 part1_info.ver
-rw-r--r-- 1 root root 219 Aug 20 2020 part2_info.ver
-rw-r--r-- 1 root root 4096 Dec 4 08:17 random_seed
-rw-r--r-- 1 root root 3 Dec 4 07:19 rxtx_mode
drwxr-xr-x 2 root root 648 Dec 4 07:20 ssh
-rw-r--r-- 1 root root 40 Dec 4 07:19 stable_secret_seed
drwxr-xr-x 3 1 root 224 Aug 20 2020 support
drwxr-xr-x 2 root root 1432 Dec 4 08:12 syslogs
-rw-r--r-- 1 root root 1 Dec 4 08:12 wips_adminstate_capwap
---------------------------------------------------------------------------
Filesystem Size Used Available Use% Mounted on
/dev/ubivol/part2 519.4M 519.4M 0 100% /part2

I dont know why its saying 0mb free at the moment though.  When I checked earlier it was 187mb free.

I had found your guide before but it didnt help me get past the EWC-AP in Recovery mode part.  As I tried the archive download-sw command but that always fails at 72.6%

[Scott Fella]

The only files I typically delete are log files and crash files.  I don't know what else you can do, what free space does the release notes tell you.  Maybe attach a screen shot of the failure or the command you are using.

[Scott Fella]

My bad... you should be using this file C9800-AP-iosxe-wlc.bin for the EWC.  On the console, try to issue capwap erase all

[peat]

Also just seen this

AP1#show filesystems
Filesystem Size Used Available Use% Mounted on
devtmpfs 938.8M 0 938.8M 0% /dev
/sysroot 940.8M 135.6M 805.2M 14% /
tmpfs 1.0M 44.0K 980.0K 4% /dev/shm
tmpfs 940.8M 0 940.8M 0% /run
tmpfs 940.8M 0 940.8M 0% /sys/fs/cgroup
/dev/ubivol/part1 371.7M 29.9M 337.1M 8% /part1
/dev/ubivol/part2 519.4M 519.4M 0 100% /part2
none 100.0M 1.5M 98.5M 1% /tmp
none 1.0M 44.0K 980.0K 4% /dev/shm
none 100.0M 1.5M 98.5M 1% /tmp/var/run/netns

Can I get into those tmpfs folders and delete things in there?

[Prince.O]

Hi Peat,

I have a few questions to better understand the issue..

1. What version of code is currently running on your 9117 AP?

- You can verify this with the following command from the AP CLI "show version".

- Please provide this output for review

2. What EWC software version are you trying to load onto the AP ? 

[peat]

Right im not sure how ive done it but ive managed to get the AP to boot back into the original firmware so the EWC-AP recovery mode has stopped.

So it looks like its back to how it was when i first plugged it in.   Its constantly in discovery by the look of it.

[*08/20/2020 00:18:38.2756] CAPWAP State: Discovery
[*08/20/2020 00:18:38.2782] Discovery Request sent to 255.255.255.255, discovery type UNKNOWN(0)

I have set a static IP on it which i was using for the TFTP  so im not sure what its doing, it also keeps rebooting after a few minutes which is exactly what it was doing when i got it out of the box.  thats why I thought (which i think was wrong)  it was a lightweight version.

the model is a 9117axi-e so is that the one with the embedded WLC already?   If so how do I access the WLC?

Ive googled a lot but keep coming up with the convert capwap to embedded WLC and that is what got my into this mess in the first place.  

To answer the what version question. this is what that shows now i have got it to boot into the original firmware.

cisco C9117AXI-E with 1928016/1410056K bytes of memory.
Processor board ID KWC24320052
AP Running Image : 8.9.100.0
Primary Boot Image : 8.9.100.0
Backup Boot Image : 17.7.1.11
Primary Boot Image Hash:
Backup Boot Image Hash: 5f0aa7bb03c338adaf25b4a937a9e92c9b96a2b1e0979dd6c5754b246fe43ad8e2a8fc3d88413ddd96bd84920f01f03ea6f6a21a6d269fb390fc9d168379009e
1 Gigabit Ethernet interfaces
2 802.11 Radios
Radio FW version : QC_IMAGE_VERSION_STRING=WLAN.HK.1.0-03095-QCAHKSWPL_SILICONZ-1.196094.2.198916.1
NSS FW version : NSS.HK.H.CS.q_len512-E_custC

Do I just need some sort of simple Command that makes it boot into EWC mode?

[peat]

More info.

I have ran the capwap erase all command and rebooted.   I have connected it up to my home network and it has picked up a dhcp IP address but the AP is flashing green blue red.   

I tried the cisco mobile app just in case to see if I could set it up that way but it cant join it.

It still looks to me like its in lightweight mode since it keeps saying 

[*08/20/2020 00:39:28.6355] CAPWAP State: Discovery
[*08/20/2020 00:39:28.6375] IP DNS query for CISCO-CAPWAP-CONTROLLER.lan
[*08/20/2020 00:39:28.6722] Discovery Request sent to 255.255.255.255, discovery type UNKNOWN(0)

but havent i got the embedded wlc model rather than the lightweight version? 

[Scott Fella]

Your output shows that the ap is running capwap. You need to extract the tat file and upload the bin file. That is the EWC controller image.  Follow the link I have posted. 

[peat]

Thats what I did in the first place which put me in recovery mode. 

I will give it another go and see if it works better the second time.

So I am guessing CAPWAP = lightweight mode? So my first assumption was right and i didnt mess the AP up?.

That will be helpful when i explain to the company who supplied the AP why I cant do the wifi survey with it on monday.

[peat]

.

[peat]

I tried again and it was going through fine then half way through the reboot I saw this

[*12/04/2021 07:20:37.1485] [error][verify_package] Fail to verify image: /part2/wlc/images/C9800-AP-iosxe-wlc.bin
[*12/04/2021 07:20:37.1485] FAILED
[ OK ] Started ME Recovery Service.
Starting ME Recovery Service...
[FAILED] Failed to start ME Install Script.
See 'systemctl status me-install.service' for details.
[*12/04/2021 07:20:37.2476] systemd[1]: Failed to start ME Install Script.
[*12/04/2021 07:20:37.2478] systemd[1]: me-install.service failed.
[ OK ] Reached target Multi-User System.
[*12/04/2021 07:20:37.2521] cat: can't open '/part2/wlc/harddisk/stable_ap_version': No such file or directory
[*12/04/2021 07:20:37.2554] cat: can't open '/part2/wlc/harddisk/stable_remote_ap_version': No such file or directory
[*12/04/2021 07:20:37.2690] cat: can't open '/part2/wlc/harddisk/stable_controller_version': No such file or directory
[*12/04/2021 07:20:37.2717] cat: can't open '/part2/wlc/harddisk/stable_remote_controller_version': No such file or directory
[*12/04/2021 07:20:37.3080] Sat Dec 4 07:20:37 UTC 2021 In Auto Recovery Mode
[*12/04/2021 07:20:37.3117] Sat Dec 4 07:20:37 UTC 2021 Checking if stable AP and WLC versions exist
[*12/04/2021 07:20:37.3154] Sat Dec 4 07:20:37 UTC 2021 No stable AP and WLC versions exist.
[*12/04/2021 07:20:37.3190] Sat Dec 4 07:20:37 UTC 2021 Checking if stable remote AP and WLC versions exist
[*12/04/2021 07:20:37.3225] Sat Dec 4 07:20:37 UTC 2021 No stable remote AP and WLC versions exist.
[*12/04/2021 07:20:37.3265] Sat Dec 4 07:20:37 UTC 2021 Attempted auto recovery, but there is no better version to switch to
EWC-AP in Recovery Mode......
Please use 'archive download-sw' to upgrade AP and Controller image.
Please use 'show flexconnect ewc-ap launch-log' to check EWC-AP launch log.

So I tried the archive download-sw command and got this

AP1#archive download-sw ewc-ap tftp://192.168.1.2/C9800-AP-iosxe-wlc.bin
Starting download eWLC image tftp://192.168.1.2/C9800-AP-iosxe-wlc.bin ...
It may take a few minutes. If longer, please abort command, check network and try again.
############## 19.7%EWC-AP in Recovery Mode......
Please use 'archive download-sw' to upgrade AP and Controller image.
Please use 'show flexconnect ewc-ap launch-log' to check EWC-AP launch log.
################################################ 67.1%EWC-AP in Recovery Mode......
Please use 'archive download-sw' to upgrade AP and Controller image.
Please use 'show flexconnect ewc-ap launch-log' to check EWC-AP launch log.
#################################################### 72.6%
curl: (23) Error
transfer command failed (Error 23)

I also tried this command

AP1#ap-type ewc-ap tftp://192.168.1.2/ap1g6 tftp://192.168.1.2/C9800-AP-iosxe-wlc.bin
Image upgrade not needed, changing AP Type to EWC-AP

So I am back to where i was at the first post.  Stuck in EWC-AP recovery mode.

[Ciphertext87]

I'm also having this problem with a C9117AXI.  I believe it was caused by a failed manual conversion from CAPWAP to EWC, there was a command syntax error between online guides and what the current installed version of code was.  This seems to have caused the upgrade to start, but fail, leaving behind an incomplete file in /dev/ubivol/part2

I've since gotten the AP to join a C9130AXI running EWC properly.  The 9117 can download the new AP image from the TFTP server the 9130 is pointing to, but it always fails to predownload the controller image (EWC).  I've monitored space availability of /part2 (/dev/ubivol/part2 is mounted as /part2) during the EWC image download process with the command "show flash cores"  I can see that the mount runs out of available space while the image download is still in progress, this seems to be due to the partial file leftover from the failed manual conversion.

I've seen posts about manually deleting the cores to resolve the issue, but it doesn't seem to work.  I haven't found a way to get into /dev/ubivol/part2 to delete the incomplete file that is using up enough space to make the image download fail.  Has anyone else figured out a solution for this issue?

[Rich R]

What version is it running and what version are you trying to convert it to?