Setting up L3 control on Nexus 1000V

In version 1.2 of the Nexus 1000V we introduced L3 based connectivity for control traffic between the VSM and its VEMs. We did this because in some environments it’s just not possible to have L2 connectivity between VSM and all the VEMs that will be attached.

 

It was still a best practice to use L2 based control but we offered L3 based connectivity for customers that needed it.

 

Going forward you will see situations where we need L3 based control over L2. Why? Well for one as we introduce integration with other products using L3 allows us to be more flexible in our deployments. L3 control is also easier to troubleshoot. You can use simple tools like ping to verify network connectivity between the VSM and VEM.

 

The downside is that it requires a VMK interface on the ESX host and an IP address. This is yet another management interface that you have to deal with.

 

The goal of this write up is to show you how to setup L3 control between the VSM and VEM.

 

The Nexus 1000V VSM VM has 3 network interfaces.

  • Control (control0)
  • Management (mgmt0)
  • Packet

 

When setting up L3 control you have two options.

  • L3 control through VSM mgmt0 interface (default)
  • L3 control through VSM control0 interface

 

Several important notes

  • When setting up L3 control both “control and packet” traffic are forwarded over the same interface.  In L3 control mode the packet nic does not get used on the VM, but don’t delete it. The VM still expects that nic to be present.
  • Also control0 and mgmt0 use a different VRF. If you use control0 you will need to setup a route in the default VRF.

n1kv-l3# show vrf interface

Interface                     VRF-Name                    VRF-ID

control0                      default                          1

mgmt0                         management                       2

 

When should I use dedicated VSM Control interface for L3?

  • If you want to isolate your control and management to the VSM VM.
  • If you want your control interface to be a in different subnet than the management interface

 

Sample topology

  • VSM is in a 172.18.217.x subnet in VLAN 2
  • My ESXi hosts are in 192.168.10.x in VLAN 10 and 192.168.11.x  in VLAN 11

 

Configure the VSM for L3 control

When running setup on a newly created VSM choose L3 when you get to the following question during setup.

setup.jpg

At this point the VSM is configured for L3 but it will be using the mgmt 0 interface.

 

n1kv-l3# show svs domain

SVS domain config:

Domain id:    43 

Control vlan: 1 

Packet vlan:  1 

L2/L3 Control mode: L3

L3 control interface: mgmt0

Status: Config push to VC failed (communication failure with VC).

 

In L3 mode control and packet vlans will show as vlan 1. No need to change these. You can also see that the L3 control interface is mgmt0.

 

Now that we have the L3 control configured on the VSM, continue with normal setup.

  • Configure VSM to vCenter connectivity
  • Configure uplink port-profiles

 

One note about your uplink port-profile: You need to create and allow the VLAN that your ESX hosts will use to communicate to the VSM.  My ESX hosts are going to communicate on VLAN 10 and VLAN 11 using a VMK interface in the 192.168.10.x and 192.168.11.x subnets. My uplink port-profile needs to look like the following

 

port-profile type ethernet uplink

vmware port-group

switchport mode trunk

switchport trunk allowed vlan 10,11,150-152

no shutdown

system vlan 10,11

state enabled

 

VLAN 10 and 11 are allowed and set as a system VLAN.

 

Next we need to create a veth port-profile for the VMK interfaces that will be used for L3 control. I need to create 2 veth port-profiles. One for hosts in VLAN 10 and another for hosts in VLAN 11.

 

N1kv-l3(config-port-prof)# port-profile type veth L3-control-vlan10

n1kv-l3(config-port-prof)# switchport mode access

n1kv-l3(config-port-prof)# switchport access vlan 10

n1kv-l3(config-port-prof)# vmware port-group

n1kv-l3(config-port-prof)# no shutdown

n1kv-l3(config-port-prof)# capability L3control

n1kv-l3(config-port-prof)# system vlan 10

n1kv-l3(config-port-prof)# state enabled

 

n1kv-l3(config-port-prof)# port-profile type vethernet L3-control-vlan11

n1kv-l3(config-port-prof)# switchport mode access

n1kv-l3(config-port-prof)# switchport access vlan 11

n1kv-l3(config-port-prof)# vmware port-group

n1kv-l3(config-port-prof)# no shut

n1kv-l3(config-port-prof)# capability L3control

n1kv-l3(config-port-prof)# system vlan 11

n1kv-l3(config-port-prof)# state enabled

 

 

Now comes the easy part. We need to make sure that ESXi hosts can ping the VSM so we know they have network connectivity.  With L2 control there was no easy way to verify network connectivity between the VSM and VEMs.

 

Next add the ESXi hosts to the Nexus 1000V in VMware vCenter. If you have VUM installed the VEM modules will automatically get loaded. If you don’t have VUM installed you will need to manually install the VEM modules on the ESXi hosts.

 

When you add the host, it will show up in VMware vCenter as added, but it will not show up as a module on the VSM until you add a VMK interface to the veth port-profile.

 

Do I need a dedicated VMK interface for control?

No. We are prefectly fine with customers migrating the mgmt vmk interface from the ESXi host to the VEM and using that interface. There have been changes on the VEM and VSM to better deal with higher latency networks and drops. Actually migrating the mgmt vmk interface makes life easier because there is no concern about needing to add static routes.

 

 

 

Create and add the VMK interface to the veth port-profile

Once you add the vmknic interface to either the L3-control-vlan10 or L3-control-vlan11 port-profile the VEM will know that it now has a way to communicate with the VSM. The VEM should show up on the VSM with a "show module" command and the VMK interface will show up with "show int virtual"

 

Note: We only support a single VMK interface per host to be assigned to the L3capability port-profile.

 

That's it for getting L3 control working with mgmt 0 on the VSM.

How about a quick example with control0

This time we will use

  • Mgmt0 172.18.217.245/24 on VLAN 2
  • Control0 192.168.150.10/24 on VLAN 150
  • ESX hosts on 192.168.10.x/24 on VLAN 10
  • ESX hosts on 192.168.11.x/24 on VLAN 11

 

Make sure control 0 NIC is on the right VMware port-group

 

vsm.jpg

Assuming a new install and you set L3 control in the setup script we pick up with configuring control 0.

 

n1kv-l3(config)# int control 0

n1kv-l3(config-if)# ip address 192.168.150.10 255.255.255.0

 

Next change the SVS domain to use control0 instead of mgmt0

 

n1kv-l3(config-if)# svs-domain

n1kv-l3(config-svs-domain)# svs mode L3 interface control0

 

See if you can ping the 192.168.150.10 interface from the ESXi hosts. You should not be able to since we have not yet set a default route for the default VRF.

 

n1kv-l3(config)# vrf context default

n1kv-l3(config)# ip route 0.0.0.0/0 192.168.150.1

 

Now add the hosts to the Nexus 1000V and you should be set.

 

You can verify with “show module” and “show module vem counters” to verify you are getting heartbeat messages between the VSM and VEM.

 

Troubleshooting:

Some quick troubleshooting tips for L3 control

 

First make sure that you have “capability L3control” in your veth port-profile that the VMK interface will be connected to. Without that the VSM does not know that L3 control is supposed to be used on that port-profile.

 

Second make sure you have the right system VLANs in the eth and veth port-profile.

 

On the ESXi hosts you can use "vemcmd show port-drops ingress/egress", "vemcmd show stats"  and "vemcmd show packets" to see if you are receiving packets on the control VMK interface. On the VSM you can run "show module vem counters" to see if you are receiving packets from the VEMs.

 

Remember that L3 controls uses UDP and port 4785 for both tx and rx. Keep this in mind for any firewalls that might be present.