Cisco ASA Anyconnect VPN Clients local IPv6 causes DNS issues

[FdeW]

Hi All,

We are experiencing some issues with different users, hope someone here can help solve it.

First our setup, we have clients connecting with Client VPN using Cisco AnyConnect version 4.9.06037 and connecting to a Cisco ASA5585-SSP-20 running  Software Version 9.12(4)2.
So this setup and the internal network is IPv4 only. The users get an IPv4 address on the Anyconnect interface and IPv4 DNS servers.
We are using Split-tunnel using an Exclude list and additionally a Dynamic Split Exclude Domains list as a Custom Attribute.

Now, since some users have both IPv4 and IPv6 in their home network or maybe somewhere else the sometimes have both v4 and v6 address on their local LAN or WLAN adapter, together with the local v4 and v6 DNS servers.
Because Windows prefers IPv6 DNS servers (I already did some extensive research into this) some users sometimes go over the internet to a certain website or parts of that website when they should go over the VPN. Most likely because they get a response from the IPv6 local LAN/WLAN DNS server as well.

To solve this i tried setting the Send all DNS lookups through tunnel setting. However, this caused some users to have other issues where service were not working at all.

The only solution i have now would be to try and give the clients an IPv6 IP address and IP as the DNS server on their Anyconnect interface as well so that takes preference over the local one.
Because the network is IPv4 only I would have to translate the requests made to this IPv6 DNS server ip to the IPv4 Internal ip of the DNS server.

Could anyone point me in the right direction to configure this?

Thanks in advance.

[Milos_Jovanovic]

Hi @FdeW,

I would try to configure client-bypass protocol option. You can read more about it here. Basically, it will instruct ASA to bypass all settings for IPv6 traffic, if it has only IPv4 traffic, and vice versa. With this one, you should configure only IPv4 on your device (remove all IPv6 configuration, if you are not actually using IPv6), and add this option under group policy.

Kind regards,

Milos

[tvotna]

I agree with @Milos_Jovanovic . "client-bypass-protocol disable" will drop all IPv6 packets and not allow them leak out to physical adapter.

After configuring "client-bypass-protocol disable", you can try Split-exclude DNS, which is available as of AC 4.10MR1. When split DNS for split exclude tunneling is configured, specific DNS queries are sent outside the VPN tunnel, to a public DNS server. All other DNS queries are tunneled to a VPN DNS server. The split-exclude DNS list is configured via custom attribute "split-dns-exclude-domains". Note that you cannot use nslookup/dig for testing, as they have their own resolver code, which is not supported by this feature.

NAT64, as you suggested, might also work, but this will make entire config crazy. Since you're going to assign IPv6 address to client VA, you'll need to do NAT64 for both DNS queries (translating AAAA to A in the request payload and A to AAAA in the response) as well as for all other user traffic going over VPN (source IP translation). You'll need to translate destination IP too (v6 to v4). This is doable and NAT chapter in the ASA config guide has something. However, inspections support is limited in NAT64 and there may be other limitations as well.

[FdeW]

Hi @Milos_Jovanovic and @tvotna,

Thanks to both of you for the responses.
Regarding the client-bypass-protocol setting, if we were to turn this on, IPv6 traffic would be allowed outside the tunnel without any control to which destinations, which would not be desirable. We are running a Tunnel Exclude at the moment so all traffic should go through the tunnel except a few exceptions.
Currently the Client-bypass-protocol setting is configured as Disabled by the default setting, however I think the local IPv6 DNS traffic is excluded from this. I think I read this somewhere but I didn't test this.

Regarding NAT64, I don't really understand yet why the DNS queries themselves need to be translated. Can't the response just be an IPv4 address? Like when I do an DNS query on our IPv4 network I can get a response with both an IPv4 and an IPv6 address in it.
So in this case the DNS request/reply communication between the client and the ASA will be IPv6 and between the ASA and the DNS server will be IPv4, however all other communication will just be IPv4.

Thanks in advance.

Regards, Frank.

[tvotna]

Hi @FdeW

I did a quick test and "client-bypass-protocol disable" works fine for me. When "client-bypass-protocol enable" is configured along with split-exclude, I indeed see in Wireshark that Windows sends DNS requests over IPv6 to physical adapter. I use "udp.port == 53 && ip.version == 6" filter to see them. If I configure "client-bypass-protocol disable" and reconnect, such requests are no longer observed. Also, if I run "ping -6 www.cisco.com", I'm getting "Ping request could not find host www.cisco.com. Please check the name and try again". So, DNS traffic is not excluded from this feature. I'm on AC4.10MR2 (4.10.02086). How many physical adapters with IPv4 and/or IPv6 assigned you have on those endpoints which experience this issue? I have only one. Maybe this is the difference or AC version matters.

For you 2nd question, you're probably right. Even more, it may not be necessary to perform any NAT at all. It appears to me that when both IPv4 and IPv6 addresses are assigned and both v4 and v6 DNS addresses are assigned, clients send three DNS requests: "DNS A request over v6 transport", "DNS A request over v4 transport" and "DNS AAAA request over v6 transport". So, 1st and 3rd will be dropped, as your corporate network is not v6-enabled, and 2nd one answered. This needs to be tested. I couldn't do this, because I'm getting AnyConnect error when I'm trying to assign v6 address to it:

Unexpected output from command 'netsh interface ipv6 set address interface="20" address="2002:0:0:0:0:0:0:1" type="unicast" store="persistent"'

[FdeW]

Hi @tvotna,

thanks again for your response and thanks for testing. So "Client-bypass-protocol disable" is configured in my configuration together with split-exclude. Did you test this with Split-DNS as well? In our configuration "Send All DNS Lookups Through Tunnel" is configured as No and DNS Names are configured.
The problem here is that the client connects to both the public and private IPv4 address of a Web Application, which are different. When we turn "Send All DNS Lookups Through Tunnel" to Yes then this problem is solved. However then we see connectivity issues on clients with both IPv4 and IPv6 addresses on the local network. 
Now, I haven't been able to verify and test this using Wireshark but I will try to do that.

As for the second part of your answer, that was my thought and that's why I configured "Send All DNS Lookups Through Tunnel" to Yes, assuming this would cause the client to only use the IPv4 DNS server and/or responses. However, as I said then we saw the connectivity issues on other clients.

Regards, Frank.

[tvotna]

I tested with split-exclude, but without tunnel-all-dns (i.e. it's by default: tunnel-all-dns disable) and without split-dns. Unfortunately, I was unable to run complete test with IPv6, as VA doesn't come up on my Windows 10 laptop if IPv6 address is assigned.

[Milos_Jovanovic]

If you want to controll IPv6 traffic, that is a perfectlly valid point, if you have IPv6 controls implemented and in place. My understanding was that you actually don't have IPv6 in place, so I don't see what are you gaining (apart from potential visibility) if you split IPv6 vs. if you hairpin IPv6 traffic (assuming that at least you can route it from your VPN GW).

I once tried to implement IPv6 setup for a customer whose VPN GW had IPv6 on outside, by doing similar setup you are talking about - split-tunnel VPN traffic, with their own IPv4 and IPv6 scopes being tunneled, and everything else being split or hairpinned, with assigning both IPv4 and IPv6 IPs for VPN clients, and I faced an issue with DNS. Clients attempted to resolve something, their ISP assigned them both IPv4 and IPv6 on local home Internet, due to Windows prefferences client used localy defined IPv6 DNS servers, which caused bunch of issues to us, so we ended up with enabling client bypass for IPv6.

I don't dare to comment on NAT64, as it is quite complex protocol and I never had any experience with it.

Kind regards,

Milos

[FdeW]

Hi @Milos_Jovanovic ,

thanks for your response. I think the below text from your comment is the same problem we are seeing. However, due to security policy we are not allowed to enable Client Bypass because this would allow IPv6 communication to everything without any control.
"and I faced an issue with DNS. Clients attempted to resolve something, their ISP assigned them both IPv4 and IPv6 on local home Internet, due to Windows prefferences client used localy defined IPv6 DNS servers, which caused bunch of issues to us"

As I stated in my reply to Tvotna I did try to configure "Send All DNS Lookups Through Tunnel" to Yes, expecting the clients to just use the IPv4 DNS servers they have on the AnyConnect interface but this caused issues with clients causing them to completely lose communication.

Regards, Frank.

[Milos_Jovanovic]

I believe it will not work that way.

If you do not cnfigure client-bypass-protocol, and you do configure some IPv6 parameters (like IPv6 pool), your client will have to use configured split-tunnel policy. Again, it is a perfectly valid point to want to control IPv6 traffic, but only if you are actuelly ready for that.

When you configure system like you did, I believe that following will happen:

• You've configured full-tunnel policy. This means that all IPv4 and IPv6 traffic will have to go via tunnel
• You've configured that all DNS traffic have to go via tunnel
• You've configured your internal IPv4 DNS servers
• You've assigned IPv6 address on your VPN device (ideally)
• Your ISP is assigning you either IPv6 or dual stack, but you do get IPv6 DNS servers
• When connected, your client is instructed to send all traffic via tunnel, both IPv4 and IPv6. Same applies for DNS, as you stated that all DNS requests must go via tunnel
• Given that your PC has IPv6 interface configured, Windows itself is forcing usage of IPv6 DNS servers, so it will try to resolve all DNS requests via IPv6 DNS
• Given that current configuration is for full-tunnel, all of your IPv4 and IPv6 traffic will be tunneled, including IPv6 DNS requests. However, these requests have nowhere to go
• Your PC will not fall back to IPv4 just because it would be more convenient

Again, you'll either have to implement IPv6 properly (including proper IPv6 routing, even if it be hairpinning traffic, IPv6 DNS servers that you'll assign, etc.), or you'll have to disable IPv6 configuration on ASA and implement client-bypass option (unless you want to disable IPv6 on clients, which is not recommended by Microsoft). I would stay away from NAT64 option.

Kind regards,

Milos

[erik8119]

We’ve been seeing similar issues with Cisco Anyconnect/Secure Client on IPv6 networks as well, and initially we thought this was caused by our deployment of Netskope for use with Netskope Private Access (NPA).

We have tried configuring the client-bypass protocol option in the Cisco client, as discussed above, to disable IPv6, and we have confirmed that it does black hole IPv6 traffic as expected, but we still see both A and AAAA DNS queries going over the wire, A via VPN and AAAA locally over IPv4 on the IPv6 enabled LAN.

What we observed in packet captures was a race condition in Windows between the A and AAAA queries, as discussed in this Fortinet article: Issues with resolving the internal FQDN w... - Fortinet Community

We changed Windows settings to prefer IPv4 over IPv6 and then disabled parallel A and AAAA queries, which resolved the issue.

Set-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip6\Parameters" -Name DisabledComponents -Value 0x20 -Type DWord

Set-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Services\Dnscache\Parameters" -Name DisableParallelAandAAAA -Value 1 -Type DWord