Re: slow data transfer rate between vlans using pbr

Wonxie · ‎04-18-2024

hi

i have cisco 6500 series core switch and facing a bizarre issue with data speed transfer.

I have two vlans svi for which is defined on this switch (core6500). these vlans are transported to another switch (sw5) via lacp etherchannel which is providing fiber uplink to two other switches at (sw19) and (sw03).and sw19 and sw03 are connected via lacp etherchannel to sw5.

I have 2 machines one on vlan 20 on sw19 and another on vlan24 on sw03. When i copy data between pcs i get hardly 2-3 Mbps speed that sometimes goes to 0kbps and then goes up to 3mbps at most.

But when i change vlan of both pcs to same vlan then i get full 1g speed for one way copy.

These vlans have PBR applied. acl of the match section of pbr says all traffic sourced from subnets of vlan20 and vlan24 towards our internal subnets is denied at the top so as to use normal routing .. all other traffic is matched by and set statement sets next hop to one of our proxy server.(10.11.101.50).

Please if someone can help me figure out it out.

interface Vlan20
ip address 10.11.20.1 255.255.255.0
no ip redirects
ip policy route-map ProxyPBR
end

!
interface Vlan24
ip address 10.11.24.65 255.255.255.192
ip policy route-map ProxyPBR
end

#sh route-map ProxyPBR
route-map ProxyPBR, permit, sequence 10
Match clauses:
ip address (access-lists): PBR-SF-INTERNET
Set clauses:
ip next-hop 10.11.101.50
Policy routing matches: 1941388 packets, 180170484 bytes
route-map ProxyPBR, permit, sequence 20
Match clauses:
ip address (access-lists): PBR-SF-WAN
Set clauses:
ip next-hop 10.11.9.1
Policy routing matches: 10637788 packets, 1191535893 bytes
route-map ProxyPBR, permit, sequence 25
Match clauses:
ip address (access-lists): PBR-WAN
Set clauses:
ip next-hop 10.11.9.10
Policy routing matches: 133926 packets, 12250584 bytes
route-map ProxyPBR, permit, sequence 30
Match clauses:
ip address (access-lists): NAT-via-R1
Set clauses:
ip next-hop 10.11.46.1
Policy routing matches: 28316 packets, 8113777 bytes

#sh ip access-lists PBR-SF-INTERNET
Extended IP access list PBR-SF-INTERNET
5 deny ip host 10.11.214.57 any
6 deny ip host 10.11.214.59 any (118 matches)
10 deny ip 10.11.20.0 0.0.0.255 10.0.0.0 0.255.255.255 (14402 matches)
15 deny ip 10.11.20.0 0.0.0.255 192.168.110.0 0.0.0.255 (13 matches)
20 deny ip 10.11.21.0 0.0.0.255 10.0.0.0 0.255.255.255 (32224 matches)
30 deny ip 10.11.22.0 0.0.0.255 10.0.0.0 0.255.255.255 (273425 matches)
40 deny ip 10.11.24.0 0.0.0.255 10.0.0.0 0.255.255.255 (384622 matches)
45 deny ip 10.11.24.0 0.0.0.255 192.168.110.0 0.0.0.255 (2 matches)
50 deny ip 10.11.26.0 0.0.0.255 10.0.0.0 0.255.255.255 (24041 matches)
51 deny ip 10.11.26.0 0.0.0.255 192.168.110.0 0.0.0.255 (3 matches)
60 deny ip 10.11.27.0 0.0.0.255 10.0.0.0 0.255.255.255 (3047863 matches)
70 deny ip 10.11.49.0 0.0.0.7 10.0.0.0 0.255.255.255
80 deny ip 10.11.49.8 0.0.0.7 10.0.0.0 0.255.255.255
90 deny ip 10.11.204.0 0.0.0.255 10.0.0.0 0.255.255.255
91 deny ip host 10.11.214.74 any (5664 matches)
100 deny ip 10.11.214.0 0.0.1.255 192.168.110.0 0.0.0.255 (13 matches)
110 deny ip 10.11.214.0 0.0.1.255 10.0.0.0 0.255.255.255 (968072 matches)
120 deny ip 10.11.215.0 0.0.0.255 10.0.0.0 0.255.255.255
130 permit ip 10.11.20.0 0.0.0.255 any (1818 matches)
140 permit ip 10.11.22.0 0.0.0.255 any (8725085 matches)
150 permit ip 10.11.21.0 0.0.0.255 any (226919 matches)
160 permit ip 10.11.24.0 0.0.0.255 any (32971099 matches)
170 permit ip 10.11.26.0 0.0.0.255 any (9449676 matches)
180 permit ip 10.11.27.0 0.0.0.255 any (3957375 matches)
190 permit ip 10.11.49.0 0.0.0.7 any
200 permit ip 10.11.49.8 0.0.0.7 any
210 permit ip 10.11.204.0 0.0.0.255 any
220 permit ip 10.11.215.0 0.0.0.255 any (14077220 matches)
230 permit ip 10.11.214.0 0.0.1.255 any (7710728 matches)
231 permit ip host 192.168.110.196 any

sh ip access-lists PBR-SF-WAN
Extended IP access list PBR-SF-WAN
5 permit ip host 10.11.214.57 10.0.0.0 0.255.255.255 log
10 permit ip host 10.11.24.83 any (8397 matches)
20 deny ip 10.11.20.0 0.0.0.255 10.11.101.0 0.0.0.255
30 deny ip 10.11.21.0 0.0.0.255 10.11.101.0 0.0.0.255
40 deny ip 10.11.22.0 0.0.0.255 10.11.101.0 0.0.0.255
50 deny ip 10.11.24.0 0.0.0.255 10.11.101.0 0.0.0.255
60 deny ip 10.11.26.0 0.0.0.255 10.11.101.0 0.0.0.255
70 deny ip 10.11.27.0 0.0.0.255 10.11.101.0 0.0.0.255
80 deny ip 10.11.49.8 0.0.0.7 10.11.101.0 0.0.0.255
90 deny ip 10.11.204.0 0.0.0.255 10.11.101.0 0.0.0.255
99 deny ip 10.11.214.0 0.0.1.255 10.11.101.0 0.0.0.255 (35 matches)
120 deny ip host 10.11.27.120 10.0.0.0 0.255.255.255 (27146 matches)
130 permit ip 10.11.20.0 0.0.0.255 10.0.0.0 0.255.255.255 (14402 matches)
140 permit ip 10.11.22.0 0.0.0.255 10.0.0.0 0.255.255.255 (273431 matches)
150 permit ip 10.11.24.0 0.0.0.255 10.0.0.0 0.255.255.255 (376231 matches)
160 permit ip 10.11.21.0 0.0.0.255 10.0.0.0 0.255.255.255 (32224 matches)
170 permit ip 10.11.26.0 0.0.0.255 10.0.0.0 0.255.255.255 (24041 matches)
180 permit ip 10.11.27.0 0.0.0.255 10.0.0.0 0.255.255.255 (3020736 matches)
190 permit ip 10.11.49.0 0.0.0.7 10.0.0.0 0.255.255.255
200 permit ip 10.11.49.8 0.0.0.7 10.0.0.0 0.255.255.255
210 permit ip 10.11.204.0 0.0.0.255 10.0.0.0 0.255.255.255
220 permit ip 10.11.215.0 0.0.0.255 10.0.0.0 0.255.255.255 (374085 matches)
230 deny ip 10.11.20.0 0.0.0.255 any (13 matches)
240 deny ip 10.11.22.0 0.0.0.255 any
250 deny ip 10.11.24.0 0.0.0.255 any (2 matches)
260 deny ip 10.11.26.0 0.0.0.255 any (3 matches)
270 deny ip 10.11.27.0 0.0.0.255 any
280 deny ip 10.11.49.0 0.0.0.7 any
290 deny ip 10.11.204.0 0.0.0.255 any
300 deny ip 10.11.215.0 0.0.0.255 any (4 matches)
310 deny ip 10.11.214.0 0.0.1.255 192.168.110.0 0.0.0.255 (9 matches)

#sh ip access-lists PBR-WAN
Extended IP access list PBR-WAN
10 deny ip any 192.168.110.0 0.0.0.255 (8209 matches)
20 permit ip host 10.11.215.21 any
30 permit ip host 10.11.215.140 any
40 permit ip host 10.11.15.20 any (10237 matches)
50 permit ip host 10.11.15.70 any (4430 matches)
60 permit ip host 10.11.15.75 any (46770 matches)
70 permit ip host 10.11.15.13 any (390 matches)
80 permit ip host 10.11.15.19 any (5463 matches)
90 permit ip host 10.11.15.18 any (2235 matches)
100 permit ip host 10.11.205.22 any
110 permit ip host 10.11.15.22 any
120 permit ip host 10.11.15.40 any
130 permit ip host 10.11.15.32 any (3080 matches)
140 permit ip host 10.11.15.21 any
150 deny ip host 10.11.46.2 10.0.0.0 0.255.255.255
160 permit ip host 10.11.46.6 any
170 permit ip host 10.11.46.3 any
180 permit ip host 10.11.46.4 any
190 permit ip any 10.11.46.0 0.0.0.7
200 permit ip host 10.11.15.16 any
210 permit ip host 10.11.15.30 any (3664 matches)
220 permit ip host 10.11.15.31 any
230 permit ip host 10.11.15.34 any (3123 matches)
240 permit ip host 10.11.15.74 any
250 permit ip host 10.11.27.120 10.0.0.0 0.255.255.255 (27146 matches)
260 permit ip host 10.11.15.12 any (4050 matches)

sh ip access-lists NAT-via-R1
Extended IP access list NAT-via-R1
10 permit ip host 10.11.214.59 any (118 matches)

MHM Cisco World · ‎04-18-2024

Sure you have slow'

The ACL of pbr is not like security ACL'

It need only permit acl to redirect traffic' other traffic by defualt use RIB' you use many deny this huge work in cpu.

Only add new ACL (keep old if this not work) with permit line only

MHM

Wonxie · ‎04-18-2024

we have a data center in same region and at different sites. so we deny these traffic from pbr.

we are using the pbr for inline content filter/proxy server. so it picks all internet traffic and all internet traffic can only be picked by using any any which will also include our data center ip address range so we used deny at the top to avoid this and let all other traffic pass via content filter/proxy server in inline mode.

regarding cpu utiilzation all the time the cpu is sleeping i.e utilization stays below 10%. its not a bottleneck

Georg Pauwen · ‎04-18-2024

Hello,

I would add these two sequences at the top of your ACLs PBR-SF-INTERNET and PBR-SF-WAN:

1 permit 10.11.20.0 0.0.0.255 10.11.24.0 0.0.0.63

2 permit 10.11.24.0 0.0.0.63 10.11.20.0 0.0.0.255

Wonxie · ‎04-18-2024

This will set next hop for traffic from vlan 20 to vlan 24 as 10.11.101.50.

where these are subnets defined on this very core switch so it should not be pbr'ed. by adding a deny for them at the top of acl i am excluding them from pbr . The reason for denying from pbr is that the pbr shold only set next hop of traffic destined for internet. All traffic destined for 10.0.0.0/8 and 192.168.110.0/24 should be routed via normal routing.

15 deny ip 10.11.20.0 0.0.0.255 192.168.110.0 0.0.0.255 (13 matches)
20 deny ip 10.11.21.0 0.0.0.255 10.0.0.0 0.255.255.255 (32224 matches)
30 deny ip 10.11.22.0 0.0.0.255 10.0.0.0 0.255.255.255 (273425 matches)
40 deny ip 10.11.24.0 0.0.0.255 10.0.0.0 0.255.255.255 (384622 matches)

MHM Cisco World · ‎04-19-2024

10% cpu is perfect but in time the packet forwarding' it jump.

Show cpu history

Debug ip policy

See howmany sec before the pbr match route-map and forward packet.

"Share debug here if you can"

MHM