PBR doesn't work on ASR1001x

[shados]

Hello everyone!

We have ASR1001x with fullview and ipsec tunnels. We use local policy to forward packets from PA addresses to their respective gateways. When the traffic load is high PBR stop working and the router starts using cef, when there is little to no traffic the PBR works fine.

When the traffic load is low:
IP: s=1.2.3.4 (local), d=5.6.7.8 (nil), len 140, local feature
UDP src=4500, dst=36682, feature skipped, NAT(2), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
IP: s=1.2.3.4 (local), d=5.6.7.8 (GigabitEthernet0/0/3.312), len 140, local feature
UDP src=4500, dst=36682, Policy Routing(3), rtype 2, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
IP: s=1.2.3.4 (local), d=5.6.7.8 (GigabitEthernet0/0/3.312), len 140, sending
UDP src=4500, dst=36682

When the traffic load is high:

FIBipv4-packet-proc: route packet from (local) src 1.2.3.4 dst 5.6.7.8
FIBfwd-proc: packet routed by adj to GigabitEthernet0/0/1.2351 9.10.11.12
FIBipv4-packet-proc: packet routing succeeded

[MHM Cisco World]

Can you share the PBR you use?

MHM

[shados]

route-map RM_LOCAL_PBR permit 10 

 match ip address ACL_SRC

 set ip vrf INTERNET next-hop 13.14.15.16

ip local policy route-map RM_LOCAL_PBR

all provider interfaces are in vrf INTERNET

[MHM Cisco World]

Next-hop is IP and you dont use verify'

So only thing make pbr not work is missing this next-hop in RIB and/or ARP can is not complete resolve the Mac of next-hop

MHM

[shados]

Why would it sometimes work and sometimes won't and why does it depend on traffic load? Any ideas?

[MHM Cisco World]

first thing are next-hop direct connect or not?

MHM

[shados]

yes, it's directly connected

[MHM Cisco World]

Show arp' did you see mac-ip or you see incomplete?

MHM

[shados]

There are no incomplete entries in arp.

[shados · Accepted solution]

I've found the root cause of the issue:

For locally generated traffic, the egress interface for non-encapsulated traffic (ISAKMP) is determined by local PBR. For locally generated traffic, the egress interface for post-encapsulated traffic (ESP) is determined by the routing tables (local PBR is not checked). For transit traffic, the egress interface for post-encapsulated traffic (ESP) is determined by the interface PBR (twice, before and after encapsulation).

[MHM Cisco World]

Thanks alot for update me

You hidden the IP' so simple Q' are the IP appear in log is same as what you use for route-map acl?

This can explain why ESP is not route via PBR but it not explain why this happened when there is high load

MHM

[shados]

Well, I actually mislead you, this issue persists regardless of the load. When I initially made packet captures there were little to no ESP packets but when I've adjusted the ACL for pcap I found them. 

[MHM Cisco World]

That explain high load' and still last Q is the ACL match Peer IP of IPsec or not?

MHM

[shados]

Yes, so for example you have an IP address of 1.1.1.1 and your PBR says nexthop is 2.2.2.2 but you run a pcap on the interface with ip addess 3.3.3.3. When I ran a pcap these 1.1.1.1 packets showed up on the wrong interface meaning PBR didn't work. This is true for ESP packets only, if you do a traceroute or ping PBR works fine.