Anyconnect client login process stuck, when DAP is enabled

ronnie.shih · ‎03-18-2024

I have a brand new pair of Cisco FTD virtual running v7.4.1 code. When DAP is enabled with hostscan scanning look for Crowdstrike AV >= v5.0 and presence of Windows domain membership registry string, the Anyconnect client gets stuck at the "Please complete the authentication process in the Anyconnect Login window" or sometimes the "Hostscan Mission complete" window. The Anyconnect login error does not time out for a long time, at least 15 to 20 minutes. I have a second pair of Cisco FTDv running the exact same code and set of DAP criteria but does not have the same issue. Hostscan being used is v4.10.08025-k9. I have verified that my SAML setup against Okta is good. The minute I take DAP off the remote access policy, everything works, hence SAML setup and remote access group policy are good.

The log that I'm able to extract from the endpoint logging in using the 'cscan.log' situated in the c:\users\username\appdata\local\cisco\Cisco HostScan\log directory shows these lines specifically at point of failure. Hostscan is actually completing, but results fail to send it appears:

[Thu Mar 14 10:55:17.530 2024][cscan]Function: log_cb_hostscan Thread Id: 0x2B04 File: c:\temp\build\thehoff\phoenix_mr80.290577643163\phoenix_mr8\posture\asa\cscan\scan.c Line: 53 Level: error :Failed in condition: opSuccess != status -> this shows that scan is completing successfully.

[Thu Mar 14 10:55:18.925 2024][cscan]Function: hs_transport_curl_post Thread Id: 0x2B04 File: c:\temp\build\thehoff\phoenix_mr80.290577643163\phoenix_mr8\posture\common\libhstransport\hs_transport_curl.c Line: 3787 Level: error :libcurl error: 56 Error
[Thu Mar 14 10:55:18.928 2024][cscan]Function: asa_post_dap Thread Id: 0x2B04 File: c:\temp\build\thehoff\phoenix_mr80.290577643163\phoenix_mr8\posture\asa\libasa\asa.c Line: 504 Level: error :results send failed; to peer (https://xx.xx.xx.xx).
[Thu Mar 14 10:55:20.165 2024][cscan]Function: asa_post_dap Thread Id: 0x2B04 File: c:\temp\build\thehoff\phoenix_mr80.290577643163\phoenix_mr8\posture\asa\libasa\asa.c Line: 514 Level: error :unable to retrieve post response.
[Thu Mar 14 10:55:21.177 2024][cscan]Function: scan Thread Id: 0x2B04 File: c:\temp\build\thehoff\phoenix_mr80.290577643163\phoenix_mr8\posture\asa\cscan\main.c Line: 986 Level: error :failed to post scan results.
[Thu Mar 14 10:55:21.182 2024][cscan]Function: halt Thread Id: 0x2B04 File: c:\temp\build\thehoff\phoenix_mr80.290577643163\phoenix_mr8\posture\asa\cscan\main.c Line: 83 Level: all :goodbye (0)

I have had a case with TAC for over a week now and it's getting nowhere so far.

ronnie.shih · ‎04-05-2024

Problem resolved, by NOT RUNNING code v7.4.1 and reverting back to v7.2.5.1

stuart-whittle · ‎05-10-2024

I seem to be running into the exact same issue. Was the downgrade recommended by TAC in the end?

ronnie.shih · ‎05-10-2024

NOT AT ALL. Cisco TAC support could not come up with an answer even with the detailed debug logs I sent them. I downgraded to v7.2.5.1 myself to try it and it worked. Now running on v7.2.6 on the latest 7.2 patch, DAP continues to work also.

Mark Ftc · ‎05-13-2024

I'm running into the "Hostscan Mission Complete" issue with an FTD running v7.4.1.1:

Its possible it might be this bug:
https://bst.cisco.com/bugsearch/bug/CSCwj08302

There seems to be a workaround - but it doesn't look like its persistent and it will have to be reapplied if the FTD reloads.

stuart-whittle · ‎05-13-2024

Just tried the workaround and it resolved my issue. Obviously not ideal that it isn't persistent. Hopefully it will be fully resolved in the next update.

ronnie.shih · ‎05-13-2024

Are you positive that's the issue? The FTDs are just ridden with a boat load of bugs and Cisco TAC wasn't even able to answer me. The manager of guy that took my ticket called and asked me to "forgive him and give him a pass" for not providing a solution to me. I found a "solution" by downgrading back down to v7.2.5.1 Other than hyperbolic and over-positive talks, let's get some real fixes.

tvotna · ‎05-13-2024

It should be very easy to verify this theory, because bug description is very detailed and workaround is provided. What I couldn't understand though is why "hostscan data-limit .." alone doesn't do its job and "debug menu ..." is needed which doesn't survive the reboot...

Symptom: After upgrading the FTD headend to version 7.4.1, AnyConnect/Secure Client is not able to establish a VPN session when HostScan/Secure Firewall Posture is enabled. The behavior is experienced as the FTD is not able to process the scanning results posted by the client when the file exceeds the 25,000 bytes. - When using the AnyConnect client with HostScan package enabled in the headend, the posture assessment process is stuck and displays the "Posture: Initiating..." message after the user successfully authenticates. - When using Cisco Secure Client with Secure Firewall Posture package enabled in the headend, the posture assessment process is stuck and displays the "HostScan mission complete" message after the user successfully authenticates. - On the VPN headend in "show counters" output below counter is increasing: ------------------ show counters ------------------ Protocol Counter Value Context ... ZERO_TRUST HUGE_PAYLOAD 354 Summary ... Conditions: - FTD version 7.4.1 - Cisco Secure Client / AnyConnect client, regardless of the version - Scanning results file exceeds 25,000 bytes (scan.xml posted by the client to the FTD headend). Workaround: 1. Navigate to the lina CLI (system support diagnostic-cli) and verify the current "hostscan data-limit" configuration by running the commands below: FTD01# show run webvpn hostscan data-limit 2. If the command above does not return any value, it means there is no hostscan data-limit set, and therefore the default value is 200kB. Run command below: FTD01# debug menu zero-trust 10 200000 2.1. If hostscan data-limit is not default and is set to a certain value, the debug menu command must match the hostscan-data limit. As shown in the example below. FTD01# show run webvpn hostscan data-limit hostscan data-limit 127000 FTD01# debug menu zero-trust 10 127000 FTD01# Note that the debug command is only valid for the duration of the FTD uptime. If the FTD headend reboots, the "debug menu zero-trust" value will default back to 25000 bytes. Therefore, the workaround will need to be applied upon every reboot.

ronnie.shih · ‎05-13-2024

So nearly 2 months after I posted this, finally someone has an answer here. No employee from Cisco responded either.

I ran into another weird bug with 7.4.1 putting a 100Mb limit on the size of the anyconnect client image you can upload. The anyconnect client image for v4.10.08025 and above exceeds 100Mb and the image upload into FMC fails. There is yet another workaround for that by manually editing a config file in expert mode at the console. I don't really understand the limit placed on these files.

Mark Ftc · ‎05-13-2024

For your AnyConnect image upload issue, take a look at this bug's notes:
https://bst.cisco.com/bugsearch/bug/CSCwi86503

Working around bugs is just an occupational hazard if you are working in IT.

Cheers

tvotna · ‎05-13-2024

Cisco engineers almost never read this forum which is understandable if you have 40+ customer cases in the backlog and get 2-3-4 new cases daily, also high-severity.

What is really bad is that "hostscan data-limit" ASA CLI is not documented, you need to use FlexConfig on FTD to change it (where is our feature parity between FTD and ASA?) and TAC was unable to tell you anything meaningful despite the libcurl diagnostics which includes error code and description. In such cases they should provide engineering image with extended diagnostic capabilities if built-in tools do not suffice.