Cisco 9300X Static routes don't work to SVI Tunnels

[Steve Adams]

NOTE: I have done the same test setup with 4000 series routers, and routing works fine.

I have successfully setup 3 SVI (Static Virtual Interface) tunnels with IPSEC between 2 9300X switches.
- I can ping the tunnel IPs
- I can ping each VLAN interface on each side of the network from the switch cli

When a computer on the VLAN pings devices on the other side of the network, it is successful through the tunnel
When the computer tries to SSH to an IP on the other side, it ignores the static route and the traffic goes out the external interface.

I see that the 9300X can implement IPSEC tunnels, but it seems they left out the routing.
- Does the 9300X Catalyst Switch not support static routes to tunnel interfaces?

==================================================
NEXUS 3172 Switch - In Between the 9300X switches
==================================================
vlan 101
name SWITCH-1
!
vlan 102
name SWITCH-2
!
interface vlan101
10.0.101.2 255.255.255.0
!
interface vlan102
10.0.102.2 255.255.255.0
!
interface Ethernet1/1
description 9300X Switch1
switchport access vlan 101
!
interface Ethernet1/2
description 9300X Switch2
switchport access vlan 102
!
!
ip route 192.168.1.1/32 10.0.101.1 (Switch-1-Loopback)
ip route 192.168.2.1/32 10.0.102.1 (Switch-2-Loopback)
ip route 10.0.1.1/32 10.0.101.1 (Switch-1-Ext-Interface)
ip route 10.0.2.1/32 10.0.102.1 (Switch-2-Ext-Interface)

9300X Switch1 - IOS XE Version 17.13.1
================================================

vlan 172
name LAN172
!
!
interface vlan 172
ip address 172.16.1.1 255.255.255.0
!
!
interface loopback0
192.168.1.1 255.255.255.255
!
!
interface Tunnel1
ip address 10.0.1.1 255.255.255.252
no ip redirects
no ip unreachables
no ip proxy-arp
tunnel source loopback0
tunnel mode ipsec ipv4
tunnel destination 192.168.2.1
tunnel protection ipsec profile IPSEC-IKEv2
!
!
interface TenGigabitEthernet1/0/1
description Switch-1-Ext-Interface
ip address 10.0.101.1 255.255.255.0
!
!
ip route 0.0.0.0 0.0.0.0 10.0.101.2
ip route 172.16.2.0 255.255.255.0 10.0.1.2

================================================
9300X Switch2 - IOS XE Version 17.13.1
================================================

vlan 172
name LAN172
!
!
interface vlan 172
ip address 172.16.2.1 255.255.255.0
!
!
interface loopback0
192.168.2.1 255.255.255.255
!
!
interface Tunnel1
ip address 10.0.1.2 255.255.255.252
no ip redirects
no ip unreachables
no ip proxy-arp
tunnel source loopback0
tunnel mode ipsec ipv4
tunnel destination 192.168.1.1
tunnel protection ipsec profile IPSEC-IKEv2
!
!
!
interface TenGigabitEthernet1/0/1
description Switch-2-Ext-Interface
ip address 10.0.102.1 255.255.255.0
!
!
!
ip route 0.0.0.0 0.0.0.0 10.0.102.2
ip route 172.16.2.0 255.255.255.0 10.0.1.2

[Steve Adams]

But when SSH is routed to the Tunnel Interface first, Loopback as source, then out the EXT interface, there is no ACL on the tunnel to go from Lan1 -to- Lan2.
- I do not want SSH to LEAK out the EXT interface, I want it to route through the tunnel

Lan1-Tun1-Loopback-Ext1 ---{routed-cloud}--- Ext2-Loopback-Tun2-Lan2

Shouldn't the traffic be routed to the tunnel interface first / encapsulated and then the ACL allows loopback -to- loopback

 permit ip 192.168.0.0 0.0.7.255 192.168.0.0 0.0.7.255  (Loopback IP -to- Loopback IP)
 permit ip 10.0.100.0 0.0.7.255 10.0.100.0 0.0.7.255 (EXT Interface -to- EXT Interface)

[MHM Cisco World]

that why I wait until I run lab by myself and add ACL to interface use for tunnel scr/dest
the issue is order, the ACL check before add tunnel header

MHM

[Steve Adams · Accepted solution]

Hello MHM,

UPDATE:  I did setup the network on my desk and routing worked fine.

I tried to simulate MTU values between 300-700 which occurs frequently with the (slow) satellite networks I work with, and I noticed that I can't even adjust the MTU on the 9300X switch below 824. I think this is my problem.
- This explains why the tunnels are UP/UP and ICMP works fine, but applications with a longer packer size don't work.
    - I am just confused why the SSH packet chose a different route than the ICMP
- I am also working with a cloud network with various SDWAN and Encryption tunnel overlay networks which complicates things.

I have come to the conclusion that the 9300X does route though the SVI tunnels as long as you have normal MTU sizes.

Thanks for all your inputs and suggestions! 
-Steve

[Steve Adams]

FYI,  My first choice was to do DM-VPN, and it is BROKEN too.
- GRE = fine
- IPSEC = Not good

The 9300X switch would actually CRASH hard and reboot. The Cisco TAC said, that DM-VPN with IPSEC is not supported.

[RAdamWilliams]

Ok, I may be misreading somewhere but from the snippets you posted it looks like your traffic (at least icmp) would absolutely go through the external interface if the tunnel was down, or even up/up but not actually passing traffic. It might help you be able to narrow down the root cause more if you only get rid of the default static route.

[Steve Adams]

Hello Adam,

You do have a good point to try and remove the default route 0.0.0.0 0.0.0.0 and test it that way.
- I really wanted the 9300X to work, but I am finding out that, just because you can configure it and the switch takes it, doesn't mean it will work.

I have a Cisco TAC case open to find out if Cisco supports it.
- My first 2 tickets with this 9300X ended with = Cisco does not support

1) DM-VPN with IPSEC = Cisco does not support = The 9300X switch actually would crash hard and reboot in a cycle = bad!
2) object-groups in ACLs = Cisco TAC case confirmed it is BROKEN

This problem with routing will be #3 = 3 strikes, you're a SWITCH = not a router!

Thanks!
-Steve

[RAdamWilliams]

Yeah, I've seen historically that beyond simple tunnels (ie no encryption, only spokes on dmvpn tunnels) you're better off using a router because you'll throw 100 commands in and one won't work but you won't know which one.

[Giuseppe Larosa]

Hello @Steve Adams ,

a switch like Cat 9300 needs to work in hardware. IF the device has no encryption hardware like some routers have, it is likely that you cannot put user traffic over IPSec tunnels because that would mean software based process switching.

The unified IOS XE makes available VPN related commands in the Catalyst 9300 but then implementation of these features may fail in the data plane.

Hope to help

Giuseppe