ISE CWA not working with 3650 running denali 16.3.6 and 16.6.4

[creserva1]

We have purchased 3650 new switches 40, configured 5 switches per stack with total of 8 stack switches to be exact and all of them comes with IOS-XE 16.3.x. but I am having issues getting all switches to work with CWA and 802.1x. 

I tried upgrading IOS-XE one stack switch from 16.3.5 to 16.3.6 still no luck. After trial and error I was able to work it with 16.6.4 I already opened support ticket but I have not heard back yet. I was wondering anyone here might have encountered this? I think it is a bug because one stack is on 16.3.6 not working and one stack is on 16.6.4 working so far with same configurations. Please note that we also have 802.1x working on 3560 IOS 15.2.

[Damien Miller]

Could you possibly give us more details around the piece that is not working?

For example, when an endpoint that you expect to hit CWA attempts to connect? Does the auth session on the port receive the CWA redirect url? Is the endpoint redirected to the portal? If redirected, after log in, do you just land back on the CWA portal, are you using DHCP relese/renew on the portal, etc?

We need to know more about your intended authentication flow in order to suggest possibly issues or potential open caveats.

[creserva1]

Both of these stack switches have same configurations only differences is stack1 16.3.6 and stack2 is 16.6.4

Stack1

STACK1#show run int gi 1/0/1
Building configuration...

Current configuration : 581 bytes
!
interface GigabitEthernet1/0/1
description B9_Client
switchport access vlan 14
switchport mode access
switchport voice vlan 96
device-tracking attach-policy IP-TRACKING
authentication periodic
authentication timer reauthenticate server
access-session port-control auto
mab
trust device cisco-phone
dot1x pae authenticator
dot1x timeout tx-period 10
auto qos voip cisco-phone
spanning-tree portfast
service-policy type control subscriber DOT1X-DEFAULT
service-policy input AutoQos-4.0-CiscoPhone-Input-Policy
service-policy output AutoQos-4.0-Output-Policy
end

STACK1#
Jul 31 2018 10:15:17.220 CST: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to up
Jul 31 2018 10:15:18.220 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up
STACK1#show access-sess int gi
Jul 31 2018 10:15:45.267 CST: %DOT1X-5-FAIL:Switch 1 R0/0: smd: Authentication failed for client (00B5.6D00.3CA9) on Interface Gi1/0/1 AuditSessionID 0A0A016200000025F0DFCAB9
Jul 31 2018 10:15:45.268 CST: %SESSION_MGR-5-FAIL:Switch 1 R0/0: smd: Authorization failed or unapplied for client (00B5.6D00.3CA9) on Interface GigabitEthernet1/0/1 AuditSessionID 0A0A016200000025F0DFCAB9
STACK1#show access-sess int gi 1/0/1 deta
STACK1#show access-sess int gi 1/0/1 details
Interface: GigabitEthernet1/0/1
IIF-ID: 0x1D49D7DA
MAC Address: 00b5.6d00.3ca9
IPv6 Address: Unknown
IPv4 Address: Unknown
User-Name: 00b56d003ca9
Status: Unauthorized
Domain: UNKNOWN
Oper host mode: multi-auth
Oper control dir: both
Session timeout: N/A
Common Session ID: 0A0A016200000025F0DFCAB9
Acct Session ID: Unknown
Handle: 0x6800001a
Current Policy: DOT1X-DEFAULT

Method status list:
Method State
dot1x Authc Failed
mab Authc Failed

STACK1#
Jul 31 2018 10:16:15.277 CST: %DOT1X-5-FAIL:Switch 1 R0/0: smd: Authentication failed for client (00B5.6D00.3CA9) on Interface Gi1/0/1 AuditSessionID 0A0A016200000025F0DFCAB9
Jul 31 2018 10:16:15.282 CST: %SESSION_MGR-5-FAIL:Switch 1 R0/0: smd: Authorization failed or unapplied for client (00B5.6D00.3CA9) on Interface GigabitEthernet1/0/1 AuditSessionID 0A0A016200000025F0DFCAB9
STACK1##unde
STACK1#undebug all
All possible debugging has been turned off
STACK1#
Jul 31 2018 10:16:45.300 CST: %DOT1X-5-FAIL:Switch 1 R0/0: smd: Authentication failed for client (00B5.6D00.3CA9) on Interface Gi1/0/1 AuditSessionID 0A0A016200000025F0DFCAB9
Jul 31 2018 10:16:45.306 CST: %SESSION_MGR-5-FAIL:Switch 1 R0/0: smd: Authorization failed or unapplied for client (00B5.6D00.3CA9) on Interface GigabitEthernet1/0/1 AuditSessionID 0A0A016200000025F0DFCAB9

*****************************************************************

Stack3

STACK3#

STACK3#show run int gi 1/0/1
Building configuration...

Current configuration : 581 bytes
!
interface GigabitEthernet1/0/1
description B9_Client
switchport access vlan 14
switchport mode access
switchport voice vlan 96
device-tracking attach-policy IP-TRACKING
authentication periodic
authentication timer reauthenticate server
access-session port-control auto
mab
trust device cisco-phone
dot1x pae authenticator
dot1x timeout tx-period 10
auto qos voip cisco-phone
spanning-tree portfast
service-policy type control subscriber DOT1X-DEFAULT
service-policy input AutoQos-4.0-CiscoPhone-Input-Policy
service-policy output AutoQos-4.0-Output-Policy
end

Jul 31 2018 10:20:22.056 CST: %EPM-6-AAA: Switch 1 R0/0: smd: POLICY xACSACLx-IP-DACL-pre-WebAuth-5b5c7f89| EVENT DOWNLOAD_REQUEST
Jul 31 2018 10:20:22.069 CST: %EPM-6-AAA: Switch 5 R0/0: smd: POLICY xACSACLx-IP-DACL-pre-WebAuth-5b5c7f89| EVENT DOWNLOAD-SUCCESS
STACK3#
Jul 31 2018 10:20:23.944 CST: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to up
Jul 31 2018 10:20:24.945 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up
STACK3#show access-se
STACK3#show access-session int gi 1/0/1 de
STACK3#show access-session int gi 1/0/1 details
Interface: GigabitEthernet1/0/1
IIF-ID: 0x117F7FC0
MAC Address: 00b5.6d00.3ca9
IPv6 Address: fe80::f47d:60a2:6a33:31ba
IPv4 Address: 10.96.14.9
User-Name: 00-B5-6D-00-3C-A9
Status: Authorized
Domain: DATA
Oper host mode: multi-auth
Oper control dir: both
Session timeout: N/A
Common Session ID: 0A0A016400000014EFDACB31
Acct Session ID: 0x0000000b
Handle: 0xf500000a
Current Policy: DOT1X-DEFAULT

Server Policies:
Vlan Group: Vlan: 14
URL Redirect ACL: ACL-WEBAUTH-REDIRECT
URL Redirect: https://10.96.50.181:8443/portal/gateway?sessionId=0A0A016400000014EFDACB31&portal=3f48ef20-7ecd-11e8-a0ec-005056859240&action=cwa&token=65bc1c45e79bc1364c26a3fed54a3930
ACS ACL: xACSACLx-IP-DACL-pre-WebAuth-5b5c7f89

Method status list:
Method State
dot1x Running
mab Authc Success

STACK3#
Jul 31 2018 10:20:51.997 CST: %DOT1X-5-FAIL: Switch 1 R0/0: smd: Authentication failed for client (00B5.6D00.3CA9) on Interface Gi1/0/1 AuditSessionID 0A0A016400000014EFDACB31
STACK3#
Jul 31 2018 10:21:03.879 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to down
STACK3#
Jul 31 2018 10:21:04.883 CST: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to down
STACK3#
Jul 31 2018 10:21:10.444 CST: %EPM-6-AAA: Switch 1 R0/0: smd: POLICY xACSACLx-IP-DACL-pre-WebAuth-5b5c7f89| EVENT DOWNLOAD_REQUEST
Jul 31 2018 10:21:10.457 CST: %EPM-6-AAA: Switch 5 R0/0: smd: POLICY xACSACLx-IP-DACL-pre-WebAuth-5b5c7f89| EVENT DOWNLOAD-SUCCESS
STACK3#
Jul 31 2018 10:21:12.173 CST: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to up
STACK3#
Jul 31 2018 10:21:13.173 CST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/2, changed state to up
STACK3#
Jul 31 2018 10:21:40.406 CST: %DOT1X-5-FAIL: Switch 1 R0/0: smd: Authentication failed for client (00B5.6D00.3CA9) on Interface Gi1/0/2 AuditSessionID 0A0A016400000015EFDB884D
STACK3#
Jul 31 2018 10:21:51.827 CST: %EPM-6-AAA: Switch 1 R0/0: smd: POLICY xACSACLx-IP-DACL-Guest_Internet-5b292354| EVENT DOWNLOAD_REQUEST
Jul 31 2018 10:21:51.859 CST: %EPM-6-AAA: Switch 5 R0/0: smd: POLICY xACSACLx-IP-DACL-Guest_Internet-5b292354| EVENT DOWNLOAD-SUCCESS
STACK3#show access-session int gi 1/0/2 details
Interface: GigabitEthernet1/0/2
IIF-ID: 0x1E0E2E0F
MAC Address: 00b5.6d00.3ca9
IPv6 Address: fe80::f47d:60a2:6a33:31ba
IPv4 Address: 10.96.14.9
User-Name: guest@domain.any
Status: Authorized
Domain: DATA
Oper host mode: multi-auth
Oper control dir: both
Session timeout: 5400s (server), Remaining: 5376s
Timeout action: Reauthenticate
Common Session ID: 0A0A016400000015EFDB884D
Acct Session ID: 0x0000000d
Handle: 0x9d00000b
Current Policy: DOT1X-DEFAULT

Server Policies:
Vlan Group: Vlan: 14
ACS ACL: xACSACLx-IP-DACL-Guest_Internet-5b292354

Method status list:
Method State
dot1x Stopped
mab Authc Success

[creserva1 · Accepted solution]

confirmed with cisco tac support that it is similar bug to https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvg07470/?reffering_site=dumpcr

They are advising to use 16.6.4 in my case, it is not a big deal since all these 40 switches came in with out-of-box of pre-installed 16.3.6 and was not on productions environment at this time. Moving to 16.6.4 I already did on 6 stack switches anyway since 802.1x is broken on 16.3.6