Cisco 9300X Static routes don't work to SVI Tunnels

[Steve Adams]

NOTE: I have done the same test setup with 4000 series routers, and routing works fine.

I have successfully setup 3 SVI (Static Virtual Interface) tunnels with IPSEC between 2 9300X switches.
- I can ping the tunnel IPs
- I can ping each VLAN interface on each side of the network from the switch cli

When a computer on the VLAN pings devices on the other side of the network, it is successful through the tunnel
When the computer tries to SSH to an IP on the other side, it ignores the static route and the traffic goes out the external interface.

I see that the 9300X can implement IPSEC tunnels, but it seems they left out the routing.
- Does the 9300X Catalyst Switch not support static routes to tunnel interfaces?

==================================================
NEXUS 3172 Switch - In Between the 9300X switches
==================================================
vlan 101
name SWITCH-1
!
vlan 102
name SWITCH-2
!
interface vlan101
10.0.101.2 255.255.255.0
!
interface vlan102
10.0.102.2 255.255.255.0
!
interface Ethernet1/1
description 9300X Switch1
switchport access vlan 101
!
interface Ethernet1/2
description 9300X Switch2
switchport access vlan 102
!
!
ip route 192.168.1.1/32 10.0.101.1 (Switch-1-Loopback)
ip route 192.168.2.1/32 10.0.102.1 (Switch-2-Loopback)
ip route 10.0.1.1/32 10.0.101.1 (Switch-1-Ext-Interface)
ip route 10.0.2.1/32 10.0.102.1 (Switch-2-Ext-Interface)

9300X Switch1 - IOS XE Version 17.13.1
================================================

vlan 172
name LAN172
!
!
interface vlan 172
ip address 172.16.1.1 255.255.255.0
!
!
interface loopback0
192.168.1.1 255.255.255.255
!
!
interface Tunnel1
ip address 10.0.1.1 255.255.255.252
no ip redirects
no ip unreachables
no ip proxy-arp
tunnel source loopback0
tunnel mode ipsec ipv4
tunnel destination 192.168.2.1
tunnel protection ipsec profile IPSEC-IKEv2
!
!
interface TenGigabitEthernet1/0/1
description Switch-1-Ext-Interface
ip address 10.0.101.1 255.255.255.0
!
!
ip route 0.0.0.0 0.0.0.0 10.0.101.2
ip route 172.16.2.0 255.255.255.0 10.0.1.2

================================================
9300X Switch2 - IOS XE Version 17.13.1
================================================

vlan 172
name LAN172
!
!
interface vlan 172
ip address 172.16.2.1 255.255.255.0
!
!
interface loopback0
192.168.2.1 255.255.255.255
!
!
interface Tunnel1
ip address 10.0.1.2 255.255.255.252
no ip redirects
no ip unreachables
no ip proxy-arp
tunnel source loopback0
tunnel mode ipsec ipv4
tunnel destination 192.168.1.1
tunnel protection ipsec profile IPSEC-IKEv2
!
!
!
interface TenGigabitEthernet1/0/1
description Switch-2-Ext-Interface
ip address 10.0.102.1 255.255.255.0
!
!
!
ip route 0.0.0.0 0.0.0.0 10.0.102.2
ip route 172.16.2.0 255.255.255.0 10.0.1.2

[MHM Cisco World]

interface Tunnel1
ip address 10.0.1.2/1 255.255.255.252
no ip redirects
no ip unreachables
no ip proxy-arp
tunnel source loopback0
tunnel mode ipsec ipv4
tunnel destination 10.0.1.1/2 <<- how tunnel destination is in same IP of tunnel? That can not be' you need to use tunnel destination the wan interface of other peer.
tunnel protection ipsec profile IPSEC-IKEv2

[Steve Adams]

Yes, I messed up the typing out the config!   I was trying to not post the production config  GOOD CATCH!
I will fix the config = they are all loopback destinations.

================================================
9300X Switch1 - IOS XE Version 17.13.1
================================================
!
interface Tunnel1
ip address 10.0.1.1 255.255.255.252
no ip redirects
no ip unreachables
no ip proxy-arp
tunnel source loopback0
tunnel mode ipsec ipv4
tunnel destination 192.168.2.1
tunnel protection ipsec profile IPSEC-IKEv2

================================================
9300X Switch2 - IOS XE Version 17.13.1
================================================

vlan 172
name LAN172
!
!
interface vlan 172
ip address 172.16.2.1 255.255.255.0
!
!
interface loopback0
192.168.2.1 255.255.255.255
!
!
interface Tunnel1
ip address 10.0.1.2 255.255.255.252
no ip redirects
no ip unreachables
no ip proxy-arp
tunnel source loopback0
tunnel mode ipsec ipv4
tunnel destination 192.168.1.1
tunnel protection ipsec profile IPSEC-IKEv2

[MHM Cisco World]

That good 

Show ip interface breif 

If the tunnel is up/up

Then only what you need is to specify the tunnel IP (10.x.x.x) in your ssh or telent' otherside the router is auto select interface and it seem that it not tunnel interface.

MHM

[Steve Adams]

Hello MHM Cisco World,

In my production, there are 4 tunnels, and they are all UP/UP.   I have truncated the config-post here to simplify what I am trying to figure out.

Basically, in a nutshell, I have isr4300 routers, and I used the same exact configuration on the 9300X switch. (IOS XE)

Pings work from switch to switch, and from device to device on the VLANs on each side. I have ACLs on the tunnels and external interfaces. Pings will not work if the tunnels were not up, ping should be blocked if it tried to go out the external interface.

So, when I try and SSH from a device on the VLAN to another device on the VLAN across the tunnel, I see a DENY on the external interface - telling me that the static route is not working.

ip route 172.16.2.0 255.255.255.0 10.0.1.2   <= This is not working

I think my question is:  Should this work?  Does the Cisco Catalyst 9300X support routing to tunnel interfaces?

[MHM Cisco World]

If it issue of routing then 

Ping form side to side  must also drop' (I assume you use source IP in ping command that reache from other peer via tunnel)

So it issue of source of packet not issue of routing 

MHM

[Giuseppe Larosa]

Hello @Steve Adams ,

try to specify also the outgoing interface

ip route 172.16.2.0 255.255.255.0 10.0.1.2 tunnel1

Hope to help

Giuseppe

[Steve Adams]

Thanks for the suggestion, I didn't think of that.

Thanks,
Steve

[Steve Adams]

The problem is, when I am on a device that is on VLAN-172 on the first switch (172.16.1.10) and I try to SSH to a device on the second switch (172.16.2.10), I see a DENY on the outgoing ACL on my external interface, so I was very confused as to why the STATIC ROUTE = "ip route 172.16.2.0 255.255.255.0 10.0.1.2" is not working. The SSH traffic should have (chosen) the route through the TUNNEL, not the external interface:

interface TenGigabitEthernet1/0/1
description Switch-1-Ext-Interface
ip address 10.0.101.1 255.255.255.0
ip access-group EXT_OUT out
ip access-group EXT_IN in

I have had a lot of problems with the 9300X switch like: object-groups do not work with ACLs, so I had to re-write all ACLs the old fashion way. And they do work that way, but, you are BLIND because counters do not work.

I feel like I have encountered another "Cisco BUG" where routing does not work through an SVI interface.
I am beginning to think that the 9300X is still a SWITCH, and routing through a tunnel is not supported.
  - But why would Cisco have tunnels work (up/up), but not routing through the tunnel?
I think the 9300X IOS XE is incomplete, the more I try to do with it, the more I find things don't work or it is BROKEN.

I am hoping it is something that I forgot and I missed.

[MHM Cisco World]

Ok, 

Then use what I usually do in dmvpn or any vpn 

Use tunnel interface (egress interface) in static route instead of using tunnel IP

MHM

[MHM Cisco World]

If this not work can you share acl you use? In and out acl

MHM

[Steve Adams]

The ACL was very BASIC. 
- Allow only HTTP/HTTPS/DNS  and DENY ALL.

NOTE: During all my testing, the web browsing and NON-TUNNEL traffic was routing/working fine.
NOTE: The IP Addresses used here are not the actual production IP's.
 - This is just for example purposes.
 - The External and Loopback interfaces are NON-PRIVATE IP's and are routable.

ip access-list extended EXT_OUT
 permit icmp any any echo
 permit icmp any any echo-reply
 permit icmp any any packet-too-big
 permit ip 192.168.0.0 0.0.7.255 192.168.0.0 0.0.7.255  (Tunnel IP -to Tunnel IP)
 permit ip 10.0.100.0 0.0.7.255 10.0.100.0 0.0.7.255 (EXT Interface -to- EXT Interface)
 permit tcp 172.16.1.0 0.0.0.255 any eq 80
 permit tcp 172.16.1.0 0.0.0.255 any eq 443
 permit udp 172.16.1.0 0.0.0.255 any eq 53
 deny ip any any log-input

ip access-list extended EXT_IN
 permit icmp any any echo
 permit icmp any any echo-reply
 permit icmp any any packet-too-big
 permit ip 192.168.0.0 0.0.7.255 192.168.0.0 0.0.7.255  (Tunnel IP Address Ranges)
 permit ip 10.0.100.0 0.0.7.255 10.0.100.0 0.0.7.255 (EXT Interface -to- EXT Interface)
 permit udp any eq 53 172.16.1.0 0.0.0.255 
 permit tcp any any established
 deny ip any any log-input

Thanks for your replies MHM!

[MHM Cisco World]

Ok, but the traffic is as following 

Inner IP add head

Scr vlan ip

Dest vlan ip

Outer IP add header

Src is tunnel source 

Dest is tunnel dest

You permit tunnel IP not tunnel Scr and tunnel dest

Can you add line to permit it and check

MHM

[Steve Adams]

Hello MHM

On the ACLs, I wanted only to permit the TUNNEL traffic from:
- LOOPBACK -to- LOOPBACK
- EXT-Interface -to- EXT-Interface

This is what is in production right now and is working fine.
INNER-VLAN traffic destined to the INNER-VLAN on the other side of the tunnel should NOT be leaked out of the external interface.

Flow:
INNER-VLAN1 <=> <IPSEC-TUNNEL/LOOPBACK> <=> EXT1 <=> EXT2 <=> <IPSEC-TUNNEL/LOOPBACK> <=> INNER-VLAN2

As I was looking at the logs, I did see a few ICMPs being denied from what looks to me, remote routers in the cloud.
- This may be Path-to-MTU-Discovery

A good question would be, if the TUNNEL is UP/UP, but the MTU was NOT adjusting because of the ICMP is being blocked from remote routers, would that effect routing so that re-directs out the default route instead of the static route?

I do see the same ICMP blocks on the isr4300 router, but it is working. So I am like  Hmmmm....

Thanks!
Steve

[MHM Cisco World]

Sorry I will continous to ask you some Q about issue' I dont think it routing issue'

Lan1-Sw1-Sw2-Lan2

If you start SSH from LAN2 to LAN1 this will failed becuase you use established keywords in ACL IN' which allow only TCP if session initiate from LAN1.

MHM