Cisco FPR1140 ikev2 site to site tunnel-FTP application dont work

[faruk.zaimovic]

Hello, 

I create Ikev2 site to site tunnel between  two cisco FPR 1140. I have same subnet on both location and i used source and destionation NAT in one site. My user try access over web some aplication and it is work, but when other site try to access first side over FTP they can access(telnet on port 21) but application dont work(they can not list directory in FTP server for example), also they can ping that server because all ports are allowed in ACP. 

FRP server work in passive mode.

Does anybody have same experience and would like to share? 

Thank you very much.

[MHM Cisco World]

Can you share the packet tracer for traffic to FTP server 

Share it for both FW and sure you need to change input interface

MHM

[tvotna]

@MHM Cisco World, just wondering, how would packet-tracer help if TCP/21 to FTP server works, but file transfer or directory listing doesn't?

@faruk.zaimovic, in general, this setup should work, so probably there is something wrong with the configuration. In passive mode client sends PASV command and the server responds with a "PASV reply" telling the client its IP address and a random TCP port to connect to. In this case firewall needs to NAT both server IP address in the IP header and a PASV reply data, so "inspect ftp" needs to be enabled. On FTD "inspect ftp" should be enabled by default, but still check "show run policy-map" and statistics in the "show service-policy inspect ftp" (check drops).

You need to somehow figure out how PASV reply is processed by the firewall and where it is sent to. Start from collecting capture with "trace detail" option on both firewalls on the interface where your FTP server lives and client lives. E.g., server side:

capture cap-in trace trace-count 100 detail interface inside match tcp host <real-server-src-ip> host <mapped-client-dst-ip>
show capture cap-in
show capture cap-in trace packet-number <N> detail

Last command might help understand what happened with the PASV reply from the server (not guaranteed).

Unfortunately, packets on the outside interface are encrypted, so troubleshooting will be difficult. Look at connection events too.

[faruk.zaimovic]

Hello,

inspect ftp is enabled i checked it. I upload pcap file in previous answer(TAC.zip) and i capture on both side traffice, and i didnt noticed anything strange. 

[faruk.zaimovic]

Hello, 

Thank you very much. I have only packet capture from both side, and in wireshark i can see that everthing is pass corectly. i upload pcap from both side and you can see. Tunnel is now down and i can not make packet tracer because i dont have traffice throught that tunnel. i reroute traffic over other firewall where i have site to site to that location.  i watch packet tracer too and it wass pass for both side. 

I think that sometining ftp block there because other traffic pass corectly.

thank you very much.

[tvotna]

PASV reply is absent in one of the .pcaps. This is the problem as I suspected. Also, I don't quite understand why 10.5.6.28 is announced in the PASV reply, while the SrcIP is 192.168.1.28. Please explain how and where exactly you collected them and what your NAT config is on each firewall (show run nat | i <corresponding rule>).

[faruk.zaimovic]

Hello, 

Thank you very much for help.

I collect packet capture in inside interface on both side.

i have to make NAT because i have same subnet at both site, and i work source and dynamic NAT for that. 10.5.6.28 is IP address that other side see.

First NAT is for 10.5.6.28 when other side tried FTP application. 

Other NAT is when first subnet access some web application on other side.

192.168.1.0/24-my local LAN(First Location)-------FPR1140----------Internet------ FPR1140-----Remote LAN(Second Location) 192.168.1.2

> show running-config nat
nat (outside,inside) source static FMF-SERVER-3-192.168.1.2 FMF-SERVER3-SEEN-FROM-FMF-10.200.200.3 destination static LFILKA-SEEN-FROM-FMF-10.5.6.28 LFILKA-192.168.1.28
nat (inside,outside) source static SVI-MO-SUBNET-192.168.0.0_16 LOCAL_LAN_MO_SUBNET-10.5.6.0_24 destination static FMF-SERVER1-SEEN-FROM-FMF-10.200.200.1 FMF-SERVER-1-172.29.235.11
nat (inside,outside) source static SVI-MO-SUBNET-192.168.0.0_16 LOCAL_LAN_MO_SUBNET-10.5.6.0_24 destination static FMF-SERVER2-SEEN-FROM-FMF-10.200.200.2 FMF-SERVER-2-192.168.1.109
nat (inside,outside) source static SVI-MO-SUBNET-192.168.0.0_16 LOCAL_LAN_MO_SUBNET-10.5.6.0_24 destination static FMF-SERVER3-SEEN-FROM-FMF-10.200.200.3 FMF-SERVER-3-192.168.1.2
nat (inside,outside) source static SVI-MO-SUBNET-192.168.0.0_16 LOCAL_LAN_MO_SUBNET-10.5.6.0_24 destination static FMF-SERVER4-SEEN-FROM-FMF-10.200.200.4 FMF-SERVER-4-192.168.5.100
nat (inside,outside) source static SVI-MO-SUBNET-192.168.0.0_16 LOCAL_LAN_MO_SUBNET-10.5.6.0_24 destination static FMF-SERVER5-SEEN-FROM-FMF-10.200.200.5 FMF-SERVER-5-192.168.5.200
nat (inside,outside) source static PARTNER-8-192.168.212.0_24 PARTNER-8-192.168.212.0_24 destination static 192.168.0.0_16 192.168.0.0_16

[tvotna]

Ok. Topology is clear now. In the fpr1140-mo-node1-capture_test.pcap capture we see "227 Entering Passive Mode (10,5,6,28,218,137).\r\n". The capture was collected on the inside interface of the firewall, i.e. before any packet processing done by the firewall. This means that the 10.5.6.28 IP in the PASV reply is incorrect. It should have been 192.168.1.28, i.e. the real IP address the server runs on. FTP inspect on the firewall sees that this IP address doesn't match transport IP address of the packet (192.168.1.28) and drops it as expected.

My opinion this is a weird FTP server misconfiguration. Check FTP server config. It doesn't need to spoof IP address in the PASV reply, because "inspect ftp" will NAT it automatically from 192.168.1.28 to 10.5.6.28 when packet is passed in the inside->outside direction.

[faruk.zaimovic]

Hello, 

Thank you for your help. I have same tunnel over cisco ASA ikev1 site to site and it application works there. when i migrate that traffic on cisco FPR1140 i have that problem. I dont have access to that application to check. Teams who works that application only told me it was working on previos device(cisco ASA ikev1), you think that something it is not ok with FTP application

[MHM Cisco World]

I still search about this issue

Indeed there is some issue with ftp in ftd' and as workaround the suggest solution is add 

Prefilter NOT ACP to allow ftp traffic.

Try add prefilter abd check pcap again.

MHM

[tvotna]

@faruk.zaimovic, I don't know how else we can explain the fact that PASV reply is absent in the client-side .pcap file and why the IP address in it is wrong in the server-side .pcap file. What you can do is to collect .pcaps again while the application is working through another path. Then we can compare them.

[MHM Cisco World]

there is retransmit after the client enter into passive mode 
so check the TCP port is is known or un-known, if it un-known you need to add it to service policy FTP inspection 
MHM

[faruk.zaimovic]

Hello, 

Thank you very much for help. In ACP I permit all ports to other side and other side permit to my side all ports.

[MHM Cisco World]

sorry for late reply but I busy this week
I re-arrange the packet you send add MAC/port source and destination and add Seq number and last the window size
as you can see the window size scale is different 
this can make issue and make one side send PSH and retransmit and in end the connection failed. 

[MHM Cisco World]

Show conn port 20 detail 

Show conn port 21 detail 

Share above from both FW 

Waiting ypur reply 

MHM