BGP null route in Cisco EVPN VxLAN fabric

[satish.txt1]

Hello,

I have very simple Cisco EVPN VxLAN fabric running on Cisco nexus 93180YC-EX switches. I am using OSFP + iBGP for underlay/overlay with Multicast. I have border-leaf switches connected to single ISP with eBGP peers and I am advertising my public subnets to ISP. 

Now I want to setup BGP Null route to stop DDoS attack. I have BGP community for RTBH from my ISP.  I have configured RTBH on border leaf and try to inject null route using tag 666 but it didn't work and I didn't see it advertise any /32 route in ISP peer. 

After google I came across with this doc [1] and they mentioned you have to configure RTBH on border-leaf + Remote leaf. I am confused why do I need to configure RTBH bgp community on border leaf? 

This is what I have currently setup on border leaf 

route-map RTBH permit 10
  match tag 666
  set community 1299:666

vrf context CUST
  description ** VRF-CUST **
  vni 10555
  ip route 85.xx.xx.128/32 Null0 tag 666
  rd auto
  address-family ipv4 unicast
    route-target both auto
    route-target both auto evpn

router bgp 65001
  vrf CUST
    log-neighbor-changes
    address-family ipv4 unicast
      redistribute static route-map RTBH
      aggregate-address 85.xx.xx.0/23 summary-only
      aggregate-address 85.xx.xx.0/24 summary-only
      aggregate-address 85.xx.xx.0/24 summary-only

I can see routes installed in table 

show ip route 85.xx.xx.128 vrf CUST
IP Route Table for VRF "CUST"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

85.xx.xx.128/32, ubest/mbest: 1/0
    *via Null0, [1/0], 11:17:20, static, tag 666

In BGP table 

2# show ip bgp vrf CUST 85.xx.xx.128
BGP routing table information for VRF CUST, address family IPv4 Unicast
BGP routing table entry for 85.xx.xx.128/32, version 3704
Paths: (3 available, best #1)
Flags: (0x880c0042) (high32 00000000) on xmit-list, is not in urib, suppressed, exported
  vpn: version 4217, (0x00000000100002) on xmit-list

  Advertised path-id 1, VPN AF advertised path-id 1
  Path type: redist, path is valid, is best path, no labeled nexthop
  AS-Path: NONE, path locally originated
    0.0.0.0 (metric 0) from 0.0.0.0 (85.xx.xx.1)
      Origin incomplete, MED 0, localpref 100, weight 32768
      Community: 1299:666
      Extcommunity: RT:65001:10555

  Path type: internal, path is valid, not best reason: Weight, no labeled nexthop
             Imported from 10.254.110.2:32869:[2]:[0]:[0]:[48]:[d89d.6769.8da0]:[32]:[85.xx.xx.128]/272
  AS-Path: NONE, path sourced internal to AS
    10.255.110.0 (metric 3) from 10.254.0.1 (10.254.0.1)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 10102 10555
      Extcommunity: RT:65001:10102 RT:65001:10555 SOO:10.255.110.0:0 ENCAP:8
          Router MAC:6cb2.aeb1.6657
      Originator: 10.254.110.2 Cluster list: 10.254.0.1

  Path type: internal, path is valid, not best reason: Weight, no labeled nexthop
             Imported from 10.254.110.1:32869:[2]:[0]:[0]:[48]:[d89d.6769.8da0]:[32]:[85.xx.xx.128]/272
  AS-Path: NONE, path sourced internal to AS
    10.255.110.0 (metric 3) from 10.254.0.1 (10.254.0.1)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 10102 10555
      Extcommunity: RT:65001:10102 RT:65001:10555 SOO:10.255.110.0:0 ENCAP:8
          Router MAC:500f.8041.ed07
      Originator: 10.254.110.1 Cluster list: 10.254.0.1

  VRF advertise information:
  Path-id 1 not advertised to any peer

  VPN AF advertise information:
  Path-id 1 not advertised to any peer

That null route didn't advertised on ISP peers

# show ip bgp vrf CUST neighbors 213.xx.xx.4 advertised-routes

Peer 213.xx.xx.4 routes for address family IPv4 Unicast:
BGP table version is 3704, Local Router ID is 85.xx.xx.1
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - best2

   Network            Next Hop            Metric     LocPrf     Weight Path
*>a85.xx.xx.0/23     0.0.0.0                           100      32768 i
*>a85.xx.xx.0/24     0.0.0.0                           100      32768 i
*>a85.xx.xx.0/24     0.0.0.0                           100      32768 i

Just wanted to confirm with expert that do I need to configure BGP community on all my remote leaf also to get null route working? 

[1] https://www.cisco.com/c/en/us/td/docs/dcn/nx-os/nexus9000/104x/configuration/vxlan/cisco-nexus-9000-series-nx-os-vxlan-configuration-guide-release-104x/m-configuring-vxlan-bgp-evpn-blackhole.html#concept_cky_r1d_jvb

[satish.txt1]

I am using route summarization do you think that could be the issue here because null route (/32) will be part of that summarization and that is why I can't see route in advertised peer? 

aggregate-address 85.xx.xx.0/23 summary-only
aggregate-address 85.xx.xx.0/24 summary-only
aggregate-address 85.xx.xx.0/24 summary-only

Does BGP null route (/32) and summary-only work together? 

[Harold Ritter]

Hi @satish.txt1 ,

The summary-only parameter clearly causes the host routes to be suppressed as these routes are part of the aggregate routes. 

Regards,

[satish.txt1]

Thank for reply, 

If I remove summary-only then my EVPN will blast /32 host routes to ISP routing table and I have almost ~2000 hosts in network with /32.  Do you think I have to remove summary-only option in order to make BGP null route to work. 

I am testing this scenario on Cisco CML lab and noticed very odd behavior that when host route /32 is installed in BGP table then BGP /32 null route isn't overriding that route and its not triggering null route. 

How do I override BGP null route on existing /32 installed route? 

Example: I have following /32 route installed in network. 

border-1-a(config-vrf)# show ip bgp vrf ISP neighbors 101.101.101.101 advertised-routes 

Peer 101.101.101.101 routes for address family IPv4 Unicast:
BGP table version is 46, Local Router ID is 69.25.124.1
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-i
njected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - b
est2

   Network            Next Hop            Metric     LocPrf     Weight Path
*>i69.25.124.0/24     10.255.255.10            0        100          0 ?
*>i69.25.124.100/32   10.255.255.10                     100          0 i

If I try to null route it won't do any effect

border-1-a(config)# vrf context ISP
border-1-a(config-vrf)# ip route 69.25.124.100/32 Null0 tag 666

border-1-a(config-vrf)# show ip bgp vrf ISP neighbors 101.101.101.101 advertised-routes 

   Network            Next Hop            Metric     LocPrf     Weight Path
*>i69.25.124.0/24     10.255.255.10            0        100          0 ?
*>i69.25.124.100/32   10.255.255.10                     100          0 i

If I pick any random IP then it will install the route in BGP table. As you can see I have added host which doesn't exist and it tigger the community blackhole. Look like EVPN BGP required some different config to deploy null route in table. 

border-1-a(config-vrf)# ip route 69.25.124.200/32 Null0 tag 666

border-1-a(config-vrf)# show ip bgp vrf ISP neighbors 101.101.101.101 advertised-routes 

   Network            Next Hop            Metric     LocPrf     Weight Path
*>i69.25.124.0/24     10.255.255.10            0        100          0 ?
*>i69.25.124.100/32   10.255.255.10                     100          0 i
*>r69.25.124.200/32   0.0.0.0                  0        100      32768 ?

. 

[Harold Ritter]

Hi @satish.txt1 ,

You can replace the "aggregate-address summary-only" with "aggregate-address suppress-map <route-map name>" to selectively suppress the more specific routes.

This allows you to only leak the more specific routes that are needed for RTBH.

Regards,

[satish.txt1]

Hmm Interesting, 

You are saying I should remove summary-only and replace aggregate-address suppress-map for all my large subnets? Sorry I never used "aggregate-address suppress-map <route-map name>" so could you please give me snippet example for my scenario. 

But other problem is in EVPN host route always advertised with /32 in routing table. If I inject /32 based null route that will not get installed in table. That is what I saw in my cisco lab. May be I am missing something here. 

[Harold Ritter]

Hi @satish.txt1 ,

> so could you please give me snippet example for my scenario. 

aggregate-address x.x.x.x/xx suppress-map example

route-map example deny 10

match ip address prefix-list RTBH-prefixes

route-map example permit 20

ip prefix-list RTBH-prefixes seq 10 permit x.x.x.x/32

Prefixes matching the prefix-list will be denied and therefore not suppressed.

> If I inject /32 based null route that will not get installed in table. 

This is not the expected behavior. The redistributed static route should be installed in the BGP table and prefer over the other BGP learnt path(s), as it is a local path. What NXOS did you run in your lab?

Regards,

[satish.txt1]

I am testing scenario on Cisco CML lab 2.5 which comes with NXOS 10.x release.  I will try your example in lab and see how it goes. 

I totally agreed that locally inject static route using blackhole community should get install in route table but its not the case with my lab. you can see example here I just insert null route but you can see in table no effect 

border-1-a(config)# vrf context ISP
border-1-a(config-vrf)# ip route 69.25.124.100/32 Null0 tag 666

border-1-a(config-vrf)# show ip bgp vrf ISP neighbors 101.101.101.101 advertised-routes 

   Network            Next Hop            Metric     LocPrf     Weight Path
*>i69.25.124.0/24     10.255.255.10            0        100          0 ?
*>i69.25.124.100/32   10.255.255.10                     100          0 i

[Harold Ritter]

Hi @satish.txt1 ,

Can you please provide the output for "sh bgp ipv4 uni 69.25.124.100/32 vrf ISP"?

Regards,

[satish.txt1]

border-1-a# sh bgp ipv4 uni 69.25.124.100/32 vrf ISP
BGP routing table information for VRF ISP, address family IPv4 Unicast
BGP routing table entry for 69.25.124.100/32, version 98
Paths: (3 available, best #3)
Flags: (0xa8080012) (high32 00000000) on xmit-list, is in urib, is backup urib r
oute, is in HW
  vpn: version 98, (0x00000000100002) on xmit-list

  Path type: redist, path is valid, not best reason: MAC Mobility Sequence, no l
abeled nexthop, is extd
  AS-Path: NONE, path locally originated
    0.0.0.0 (metric 0) from 0.0.0.0 (69.25.124.1)
      Origin incomplete, MED 0, localpref 100, weight 32768
      Community: 1299:666 
      Extcommunity: RT:65001:10555

  Path type: internal, path is valid, not best reason: Weight, no labeled nextho
p
             Imported from 10.255.1.11:32867:[2]:[0]:[0]:[48]:[5254.0008.df8b]:[
32]:[69.25.124.100]/272 
  AS-Path: NONE, path sourced internal to AS
    10.255.255.10 (metric 81) from 10.255.0.1 (10.255.0.1)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 10100 10555
      Extcommunity: RT:65001:10100 RT:65001:10555 SOO:10.255.255.10:0 ENCAP:8
          Router MAC:5211.8d05.1b08
      Originator: 10.255.1.11 Cluster list: 10.255.0.1 

  Advertised path-id 1, VPN AF advertised path-id 1
  Path type: internal, path is valid, is best path, no labeled nexthop, in rib
             Imported from 10.255.1.12:32867:[2]:[0]:[0]:[48]:[5254.0008.df8b]:[
32]:[69.25.124.100]/272 
  AS-Path: NONE, path sourced internal to AS
    10.255.255.10 (metric 81) from 10.255.0.1 (10.255.0.1)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 10100 10555
      Extcommunity: RT:65001:10100 RT:65001:10555 SOO:10.255.255.10:0 ENCAP:8
          MAC Mobility Sequence:00:1 Router MAC:520b.8335.1b08
      Originator: 10.255.1.12 Cluster list: 10.255.0.1 

  VRF advertise information:
  Path-id 1 advertised to peers:
    101.101.101.101

  VPN AF advertise information:
  Path-id 1 not advertised to any peer

[Harold Ritter]

Hi @satish.txt1 ,

The reason the local path is not selected is that the remote path has a MAC mobility sequence extended community. This causes the remote path to be selected over the local one.

In your initial post, you included the output where the remote path does not have the MAC mobility sequence extended community and the local path is selected.

BGP routing table entry for 85.xx.xx.128/32, version 3704
Paths: (3 available, best #1)
Flags: (0x880c0042) (high32 00000000) on xmit-list, is not in urib, suppressed, exported
  vpn: version 4217, (0x00000000100002) on xmit-list

  Advertised path-id 1, VPN AF advertised path-id 1
  Path type: redist, path is valid, is best path, no labeled nexthop
  AS-Path: NONE, path locally originated
    0.0.0.0 (metric 0) from 0.0.0.0 (85.xx.xx.1)
      Origin incomplete, MED 0, localpref 100, weight 32768
      Community: 1299:666
      Extcommunity: RT:65001:10555

Regards,

[satish.txt1]

Hi,

My initial post is my production network and recent one is my Cisco CML lab to mimic environment to POC. 

Hmm I have to google to understand what is MAC Mobility and what is the differences between my initial post and recent one. Where is that MAC mobility coming from and is it what I need to advertise blackhole community to my ISP for null routing host? fyi, in production I do have summary-only flag set in BGP config because I can't take it out without run some test in my Cisco LAB. 

I will run some test in lab and get back to you with your advice. 

I really appreciate your help and thank you for taking sometime out of your day to help me. 

[satish.txt1]

Hi Harold,

Quick question, as you mentioned in my initial post I don't have MAC mobility that is why it select local path. But I can't see it in advertised peer routes because I have summary-only configured in production. If I changed summary-only with suppress-map then my problem will get resolved correct? 

This is what I have in production. 

vrf CUST1
    log-neighbor-changes
    address-family ipv4 unicast
      redistribute static route-map RTBH
      aggregate-address 85.xx.96.0/23 summary-only
      aggregate-address 85.xx.98.0/24 summary-only
      aggregate-address 85.xx.99.0/24 summary-only

Because of summary-only I can't see my null routed host IP right?

# show ip bgp vrf VIVOX neighbors 213.xxx.xxx.4 advertised-routes

   Network            Next Hop            Metric     LocPrf     Weight Path
*>a85.xx.96.0/23     0.0.0.0                           100      32768 i
*>a85.xx.98.0/24     0.0.0.0                           100      32768 i
*>a85.xx.99.0/24     0.0.0.0                           100      32768 i

 But I can see in route info that my best path is locally installed prefix path 

BGP routing table entry for 85.xx.xx.128/32, version 3704
Paths: (3 available, best #1)
Flags: (0x880c0042) (high32 00000000) on xmit-list, is not in urib, suppressed, exported
  vpn: version 4217, (0x00000000100002) on xmit-list

  Advertised path-id 1, VPN AF advertised path-id 1
  Path type: redist, path is valid, is best path, no labeled nexthop
  AS-Path: NONE, path locally originated
    0.0.0.0 (metric 0) from 0.0.0.0 (85.xx.xx.1)
      Origin incomplete, MED 0, localpref 100, weight 32768
      Community: 1299:666
      Extcommunity: RT:65001:10555

Am i correct in my analysis? If I use suppress-map in production setup then everything will work as I want and IP will get null route, right? 

[Harold Ritter]

Hi @satish.txt1 ,

> Am i correct in my analysis? If I use suppress-map in production setup then       > everything will work as I want and IP will get null route, right?

That is correct.

Regards,

[satish.txt1]

I will try it out in my production and get back to you with result. I have one more question about your example. You have statically use /32 in prefix-list to no suppress route. I would like to have that some automated. For example I have Fastnetmon tool collecting sflow and detecting DDoS target and sending IP prefix route for null route target IP with proper community string using goBGP program. If I want to achieve kind of scenario in that case what kind of prefix list i should craft?