Possessed Firewall?

[mikedeyoung]

So here's a really unusual problem for which I would love a logical explanation. Is my Firewall ASA software infected with Malware?!?

So a couple weeks ago my customer calls me complaining of a problem at the corp office. They have a Service-Provider (Frontier) broadband fiber internet connection coming in which cables to a Catalyst 3560 and out of the 3560 into Firewall_A and Firewall_B. Topology Looks like this...

Internet - 3560 - Firewall_A (MAC=d3:68, IP=.4)
                         - Firewall_B (MAC=31:b7, IP=.6)

Firewall_A provides mobile Anyconnect VPN clients access to internal network. Firewall_B provides some server port-forwards and a branch office L2L IPSEC VPN access to interal corp network.

This topology has been in production for over a year. No problems up until a couple weeks ago.

So the complaint is that, intermittently, Firewall_A "stop working". After a power-cycle, it seems to work fine for a while and then it stops working again.

To make a very long story short, it turns out that when the Service-Provider Gateway (Juniper, MAC=f5:c7, IP=.1) sends out an ARP for the IP for Firewall_A, BOTH Firewall_A and Firewall_B reply. In cases where Firewall_B replies AFTER Firewall_A, the Service-Provider gateway caches an incorrect IP to MAC and traffic intended to reach Firewall_A gets delivered to Firewall_B.

The attached pic is a partial Wireshark screenshot of a capture showing Firewall_B responding to an ARP request for which it should NOT be.

Section_A = a successful ICMP Echo Request originated by a device out on Internet (x.x.x.160) reaching Firewall_A (x.x.x.4) and the Echo Reply.

Section_B = the Service-Provider ARP intended to be answered by device with x.x.x.4 assigned and being answered by Firewall_A and Firewall_B!!!

Section_C = failing ICMP Echo Request originated by device out on the Internet (x.x.x.160) because the desintation MAC is incorrect!

Also attached is a config of Firewall_B and two screenshots showing the output of "show int" for Firewall_A and Firewall_B.

*** Why is Firewall_B responding to ARP requests for which it does NOT have an interface configured for the IP in the ARP?!?!?! ***

[johnd2310]

Hi,

Could be proxy-arp on the asa causing the issue. Have you tried disabling proxy-arp on the outside interface of the firewalls?

Thanks

John

[Marius Gunnerud · Accepted solution]

Have you provided a full output of your running configuration?

Check xlate also to see if you have an entry for the other firewall public IP there.  For proxy-arp to happen the ASA needs a NAT statement for that public IP.

Next time the issue happens, if you issue the command clear conn all, is the issue corrected?

[mikedeyoung]

So this was an interesting issue.

It appears the following NAT statement, used for L2L IPSEC NAT exemption, was causing my dilemma...

nat (inside,outside) source static ALL_ADDRESS_SPACE ALL_ADDRESS_SPACE destination static OUTSIDE_ADC_TOLEDO_STAFF OUTSIDE_ADC_TOLEDO_STAFF.

The following resolved the issue...

nat (inside,outside) source static ALL_ADDRESS_SPACE ALL_ADDRESS_SPACE destination static OUTSIDE_ADC_TOLEDO_STAFF OUTSIDE_ADC_TOLEDO_STAFF no-proxy-arp

(The reason for ALL_ADDRESS_SPACE) is because no split-tunnel at remote branch office).

What is quite strange is that this config worked fine in production for a very long time without any issues.

The link below provides more explanation.

https://www.cisco.com/c/en/us/support/docs/security/adaptive-security-appliance-asa-software/116154-qanda-ASA-00.html

Thx Marius Gunnerud