I have been working on an environment where a design was completed prior to my engagement. This is my first foray into configuring VXLAN but I have some experience with overlays/ACI/etc so am fairly comfortable. Design has 2x border leafs, 2x spines, 2x leafs. It has numerous VLANs and VRFs. Lots of external routing on the border leafs. No multi-site connectivity (full L3 segmentation between sites).
Currently we are mid-migration and are having Layer 2 reachability issues on one specific VLAN with a silent host on the old network fabric not being able to communicate with VLAN-adjacent hosts on the new fabric. All other inter-fabric communications are working fine (both L2/L3).
Some basic setup notes:
- Distributed Anycast GW is enabled (i.e. fabric forwarding anycast-gateway-mac xxxx.xxxx.xxxx + (at SVI level) fabric forwarding mode anycast-gateway)
- Host learning is being done via BGP/EVPN
- ARP Suppression is enabled per VLAN
interface nve1
no shutdown
host-reachability protocol bgp
source-interface loopback1
member vni 200002
suppress-arp
mcast-group 239.0.1.1
member vni 200003
suppress-arp
mcast-group 239.0.1.5
!
interface Vlan2
no shutdown
mtu 9216
no ip redirects
ip address 192.168.1.1/24
no ipv6 redirects
ip pim sparse-mode
ip pim neighbor-policy NONE*
fabric forwarding mode anycast-gateway
Some thoughts I wanted someone to comment on:
- With a Distributed Anycast gateway SVI, should I have it configured on both border leaf and leaf nodes if end-hosts are connected to both nodes types? The design has Anycast GW either on leaf or border leaf but not both. During migration we have hosts connected to both. At end-state nodes for each VLAN won't be connected to both types. I suspect the Anycast GW configuration matches this "end-state" scenario but won't work for the migration phase.
- With a silent host (i.e. one that doesn't communicate unless it is specifically spoken to), is it best-practice to disable ARP suppression on the associated VLAN the silent-host lives on? My thinking here is that ARP suppression in combination with EVPN host-learning just doesn't work when a silent host is in the mix. More context on silent host situation below...
Silent Host Issue Notes:
- FHRP GW IP for VLAN on old network still.
- Some hosts on VLAN are connected to new VXLAN fabric but can't communicate with silent hosts on old fabric (all Layer 2 traffic).
- On border leaf, if I configure SVI (non-GW) IP to be on assocaited VLAN and ping silent host the VXLAN fabric updates to have reachability for silent host and all is well.
- The border leaf is where old fabric connects to new fabric (Layer 2 connection between).
- Layer 2 forwarding works for other VLANs in this situation, it's just this silent host that isn't playing nice until I configure a gateway and manually setup the ping.
