unknown IPs in FTD outbound logs

[tato386]

I am managing FTD-1120s with FMCv both running v7.2.1 software.  In my ACP I have a GEO rule to block all outbound traffic to China, Russia, and I few other "hotspots".   When I search for events that match this rule, most of the traffic is from internal private IPs going out on TCP/443 and these connections are blocked as expected.   However, I also see many connections for IPs that are not internal private IPs and neither do they belong to any of our public IP blocks.  How do these packets from IPs that are not on any of our networks being routed or otherwise somehow making it to our FTD public interfaces?  Sometimes the FTD tries to route the packets from one ISP interface to another ISP interface but weirder still sometimes the "egress" interface is one of our internal interfaces.   So why would the FTD want to route some random and unknown IP to an internal interface?  I have attached a sample with our internal interface redacted for privacy.

TIA,

[Aref Alsouqi]

That is very interesting and I would suggest raising it with TAC as it seems to be a buggy behaviour.

[tato386]

yeah, definitely gonna do that.  thx

[MHM Cisco World]

I will try help you to detect the issue' but sorry if my reply is take time' I am busy this day.

Anyway' 

Are you use any proxy ?

MHM

[tato386]

 I do not use proxies, but I do use PBR to override default gateway and send certain IPs/subnets out the 2nd ISP using an ACL and Flexconfig.

BTW, I appreciate any help you can give me so don't about how long it takes.  

Thanks

[MHM Cisco World]

Select IP appear in event view'

Use packet-tracer and see if the route-lookup is use pbr or RIB

MHM

[tato386]

The traces do not seem consistent with event logs on FTD.  I tested with IPs that are shown in the FTD logs as egress being internal interface but trace shows ISP is the egress.  The trace makes more sense than FTD log because it makes sense the packet would be routed to the default gateway and not towards the inside.  I do not see any PBR being applied. 

I also see the inbound packets initially being allowed, then dropped as they try to egress.  I guess this makes sense because the FTD needs to ingest the packet and pass it thru the SNORT engine in order to do the GEO check before it realizes it needs to block it.  Maybe the trace is showing what would have happened to the packet had it not been blocked by SNORT GEO rule?

What still worries me is why/how does traffic with a destination IP that does not belong to me get routed to my public interfaces?  I would understand one ISP having some routing issues but both??

[tato386]

so it gets weirder still.  working on the issue with Cisco TAC we find event logs showing unknown foreign IPs entering the FTD from an internal zone/interface?!  TAC says packets are being blocked and packet tracer seems to confirm this.  TAC says its normal activity being logged incorrectly. Might be a bug but as of now, nothing fitting this behavior has been logged as a bug.   

So maybe there is nothing to worry about but still this is something that doesn't exactly give me the warm fuzzies.  I kindly asked them to start a bug report.  The attachment shows the foreign IPs bouncing around inside my FTD, being routed in and out.  weird...

[MHM Cisco World]

You use IGP BGP or default route in this FTD?

MHM

[tato386]

no dynamic routing at all on this box.  just two static default routes using PBR to route specific internal IPs out using their assigned ISP.

[MHM Cisco World]

I think alot About your issue' 

The FTD receive traffic abd need to forward it.

But Q here why ISP send this traffic toward your FTD out1 interface even if the public IP is different than interface use?

Mostly it issue of ISP routing' your FTD look like transit device.

MHM

[tato386]

yes, I agree about the transit thing but it brings up many questions.  What is causing the ISP to send this traffic to my device?   Why does FTD even bother to try to route it?  What's worse is that it passes what should be un-routable traffic to the SNORT engine taking up valuable resources.  Seems like a good way to run a DoS attack.   And of course, then we have the issue of some of this traffic coming in on what are inside, private interfaces....

[MHM Cisco World]

for ISP why it routing traffic toward you' that you need to check with ISP.

For FTD you mentioned that it have defualt route so it send to snort not log as un-routable.

Again FTD do job of inspect traffic' but why this traffic hit your Out interface need to talk with ISP' I think they have answer for that

MHM

[tato386]

I don't think I have time to deal with ISP but I will follow up with TAC on traffic on inside and post back.  Thanks