Re: "VEM SOURCE IP NOT CONFIGURED" for ERSPAN?

jiangcaixia · ‎05-11-2010

I have some troubles when I configure ERSPAN feature on nexus1kv.

The environment i configured is:

1. There are two subnets: 10.112.* and 10.117.*

2. There is one nexus1kv in each subnet. Each nexus1kv is managed by one vc.(one vc in each subnet)

I want to export the packets from 10.112.* to the interface in 10.117.*. I new a port profile with "capability l3control" on nexus1kv in 10.112.* subnet. And go to vSphere Client and configure a VMKNIC for the ESX Host to make sure it points to this port profile as a new virtual adapter.

Then, I create a monitor session with source interface in 10.112.* and destination interface in 10.117.*. And I use netperf to create the packet flow for the interface in 10.112.*. But I can't get the duplicated packets from 10.112.* in 10.117.*.

So, I wonder whether there is some wrong configuration.

Can I use two vc to use ERSPAN?

Any idea?

Thanks,

Caixia

jiangcaixia · ‎05-11-2010

The host for nexus1kv is esx with service console.

The configuration of ERSPAN is:

Nexus1kv(config)# show monitor session 1
   session 1
---------------
description       : erspan
type              : erspan-source
state             : up
source intf       :
    rx            : Veth1
    tx            : Veth1
    both          : Veth1
source VLANs      :
    rx            :
    tx            :
    both          :
filter VLANs      : filter not specified
destination IP    : 10.117.4.49   // this is the ip for an ubuntu machine running wireshark, Can the destination IP be the virtual ethernet interface of Nexus1kv?
ERSPAN ID         : 55
ERSPAN TTL        : 64
ERSPAN IP Prec.   : 0
ERSPAN DSCP       : 0
ERSPAN MTU        : 1500
ERSPAN Header Type: 2

Nexus1kv(config)# show port-profile usage

-------------------------------------------------------------------------------
Port Profile               Port        Adapter        Owner
-------------------------------------------------------------------------------
system-uplink              Eth3/2      vmnic1         10.112.120.64
vm-pg                      Veth1       Net Adapter 1 testvds
vm-pg-erspan               Veth3       vmk0           Module 3
vm-pg2                     Veth2       Net Adapter 1 testvds-ubuntu

Nexus1kv# module vem 3 execute vemcmd show span

VEM SOURCE IP NOT CONFIGURED.

HW SSN ID DST LTL/IP ERSPAN ID HDR VER
0 10.117.4.49 55 2

On 10.112.120.64, "vmkping 10.117.4.49" is ok. Why the above command shows "VEM SOURCE IP NOT CONFIGURED"?

Any ideas? Thanks.

Caixia

jiangcaixia · ‎05-12-2010

Does anyone use ERSPAN before? Do you have any ideas?

Thanks in advance！

Caixia

Robert Burns · ‎05-13-2010

Are you sure you set the IP address, Mask & GW for the VMKernel Interface assigned to the ERSPAN Port Profile?

Can you please past the config of

1. ERSPAN port profile.

2. VSM output of: module vem 3 execute vemcmd show span

3. From the CLI of your ESX host (VEM 3) esxcfg-vmknic -l

Here's what a typical ERSPAN (for SPAN of all uplink traffic) would look like:

nexus1kv(config)# monitor session 1 type erspan-source

nexus1kv(config-erspan-src)# source interface ethernet 3/3
nexus1kv(config-erspan-src)# destination ip 10.54.54.1
nexus1kv(config-erspan-src)# erspan-id 999
nexus1kv(config-erspan-src)# mtu 1000
nexus1kv(config-erspan-src)# no shut

nexus1kv(config)# show monitor session 1
   session 1
---------------
type              : erspan-source
state             : up
source intf       :
    rx            : Eth3/3
    tx            : Eth3/3
    both          : Eth3/3
source VLANs      :
    rx            :
    tx            :
    both          :
filter VLANs      : filter not specified
destination IP    : 10.54.54.1
ERSPAN ID         : 999
ERSPAN TTL        : 64
ERSPAN IP Prec.   : 0
ERSPAN DSCP       : 0
ERSPAN MTU        : 1000

nexus1kv # module vem 3 execute vemcmd show span

VEM SOURCE IP: 10.54.54.10

HW SSN ID            DST LTL/IP                 ERSPAN ID
        0                    47                 local
        1                   10.54.54.1          999

Regards,

Robert

jiangcaixia · ‎05-13-2010

Hi Robert,

Thanks for your reply. I still get "VEM SOURCE IP NOT CONFIGURED". The following is my answer to your questions.

Are you sure you set the IP address, Mask & GW for the VMKernel Interface assigned to the ERSPAN Port Profile?

[Caixia] Yes, the ip address/mask/gw are assigned by DHCP. "esxcfg-vmknic -l" displays the result.

1. ERSPAN port profile.

port-profile vm-pg-erspan
description:
type: vethernet
status: enabled
capability l3control: yes
pinning control-vlan: -
pinning packet-vlan: -
system vlans: none
port-group: vm-pg-erspan
max ports: 32
inherit:
config attributes:
    switchport mode access
    no shutdown
evaluated config attributes:
    switchport mode access
    no shutdown
assigned interfaces:
    Vethernet3

2. VSM output of: module vem 3 execute vemcmd show span

Nexus1kv# module vem 3 execute vemcmd show span

VEM SOURCE IP NOT CONFIGURED.

HW SSN ID DST LTL/IP ERSPAN ID HDR VER
0 10.117.4.49 55 2

3. From the CLI of your ESX host (VEM 3) esxcfg-vmknic -l

[root@localhost ~]# esxcfg-vmknic -l
Interface Port Group/DVPort IP Family IP Address Netmask Broadcast MAC Address MTU TSO MSS Enabled Type
vmk0 224 IPv4 10.112.120.104 255.255.252.0 10.112.123.255 00:50:56:75:64:20 1500 65535 true DHCP

4. esxcfg-vswitch -l

[root@localhost ~]# esxcfg-vswitch -l
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch0 128 6 128 1500 vmnic0

PortGroup Name        VLAN ID Used Ports Uplinks
Control               3000     1           vmnic0
VM Network            0        1           vmnic0
Packet                3002     1           vmnic0
Service Console       0        1           vmnic0

DVS Name Num Ports Used Ports Configured Ports MTU Uplinks
Nexus1kv 256 53 256 1500 vmnic1

DVPort ID           In Use      Client
160                 1           vmnic1
161                 0
162                 0
163                 0
164                 0
165                 0
166                 0
167                 0
168                 0
169                 0
170                 0
171                 0
172                 0
173                 0
174                 0
175                 0
176                 0
177                 0
178                 0
179                 0
180                 0
181                 0
182                 0
183                 0
184                 0
185                 0
186                 0
187                 0
188                 0
189                 0
190                 0
191                 0
224                 1           vmk0
192                 1           testvds-ubuntu.eth0
128                 1           testvds.eth0
193                 0

Thanks,

Caixia

prvc · ‎05-18-2010

In your ERSPAN port-profile, there is no VLAN defined and hence there is no system VLAN defined. This is a must for ERSPAN to work. Can you repeat the activity after you do that?

Praveen

jiangcaixia · ‎05-18-2010

Hi Praveen,

Thanks for your reply.

Do you mean that I should use "switchport access vlan vlan_id" and "system vlan vlan_id" in ERSPAN port-profile? Which num can I use for this vlan_id?

I want to explain my environment for nexus1kv: I only use two vlans for control vlan and packet vlan. So I don't know which vlan_id I should use for ERSPAN port-profile. And if I set a random num for this vlan_id, it will get the following error:

Nexus1kv(config-port-prof)# switchport access vlan 2
Nexus1kv(config-port-prof)# no shutdown
Nexus1kv(config-port-prof)# system vlan 2
ERROR: Some of the input vlans are not active.
All system vlans should be in the active state

Do you have any more suggestions?

Thanks,

Caixia

prvc · ‎05-18-2010

Hi Caixia,

You are getting that error while configuring VLANs because you have not configured that VLAN in your N1K. You can configure any VLAN as your ERSPAN VLAN, the only prerequisite is that it must be globally routable from your upstream switch.

Let us say you are using VLAN 10 for your ERSPAN. Create the ERSPAN port-profile in access mode with VLAN 10. Make VLAN 10 as system vlan in that port-profile. Now VLAN 10 must be configured (made active) in both N1K and your upstream switch. Make sure that you allow the same in your uplink port-profile and the corresponding port in the upstream switch. Now create an L3 interface for this VLAN in your upstream switch and assign an ip address for it. Now make sure that this ip address is routable to your destination 10.117.* network. Now make sure that the vmknic ip address is in the same subnet as VLAN 10. Try end to end connectivity using a vmkping from your host. Now ERSPAN must work.

Regards

Praveen

jiangcaixia · ‎05-18-2010

Hi Praveen,

Thanks. I have some confusions about nexus1kv usage.

Because I have no physical switch to support nexus1kv. So I just use one host to configure nexus1kv. The vsm is in a vm on this host. And I think the control vlan and the packet vlan are necessary vlans for nexus1kv to make communications between vsm and vem. So, in my environment, vsm and vem are on the same machine, therefore, I think I don't need the support of the physical switch. I create the uplink port-profile to support system vlans(control vlan and packet vlan) and the regular port-profile without the vlan setting. So, in this case, I can make my data vm installed on these regular port-profile to connect to the outside network(with no vlan setting). And the vsm can still make communication to vem.

So, in this situation, how can I use ERSPAN?

The following are some my configurations:

port-profile system-uplink
description:
type: ethernet
status: enabled
capability l3control: no
pinning control-vlan: -
pinning packet-vlan: -
system vlans: 3000,3002
port-group: system-uplink
max ports: -
inherit:
config attributes:
    switchport mode trunk
    switchport trunk allowed vlan all
    no shutdown
evaluated config attributes:
    switchport mode trunk
    switchport trunk allowed vlan all
    no shutdown
assigned interfaces:
    Ethernet3/2

port-profile vm-pg
description:
type: vethernet
status: enabled
capability l3control: no
pinning control-vlan: -
pinning packet-vlan: -
system vlans: none
port-group: vm-pg
max ports: 32
inherit:
config attributes:
    switchport mode access
    no shutdown
evaluated config attributes:
    switchport mode access
    no shutdown
assigned interfaces:
    Vethernet1

Nexus1kv(config)# show port-profile usage

-------------------------------------------------------------------------------
Port Profile               Port        Adapter        Owner
-------------------------------------------------------------------------------
system-uplink              Eth3/2      vmnic1         10.112.120.64
vm-pg                      Veth1       Net Adapter 1 testvds
vm-pg-erspan               Veth3       vmk0           Module 3

I also have other questions about ERSPAN:

1. why ERSPAN port-profile needs a vlan_id. Does this vlan_id will be encapsulated into the ERSPAN packet?

2. What's the format of these ERSPAN packets if I can capture them on the destination port? Is the source mac address the vmknic ip address for ERSPAN port-profile? Is the destination mac address "10.117.4.49" which I set in ERSPAN monitor session?

Thanks in advance!

Regards,

Caixia

prvc · ‎05-18-2010

Hi Caixia,

I can see that in the upstream port-profile you have allowed all vlans and in the vethernet port-profile you have not specified any VLANs explicitly. That means that all your vethernet ports will be in VLAN 1 by default. I am wondering how you are using two different subnets an sending packets between them without a routing device?

Now to make ERSPAN work you can try the following:

1. Declare VLAN 1 explicitly in N1K and make it active

2. In the ERSPAN port-profile explicitly mention VLAN 1 and make that VLAN as system VLAN

3. In the vmknic give an ip address in the 10.117* range instead of 10.112* range.

Now for your questions:

1. The VLAN id will not be encapsulated along with the ERSPAN packet, but we mention a VLAN because the ERSPAN module in N1K is designed n such a way that now it needs a VLAN and needs to be a system VLAN to work.

2. The ERSPAN packets will be normal IP packets which will be encapsulated using GRE and ERSPAN header. You can capture this at your destination and check. The source IP for this packet will be ip address of the vmknic of the host from which you are ERSPANing. The destination will be destination ip you have mentioned in the ERSPAN configuration.

Regards

Praveen

jiangcaixia · ‎05-18-2010

Thanks, Praveen.

I have captured the ERSPAN packets after I configured 1&2 to explicitly set vlan 1 in nexus1kv and ERSPAN port-profile. Thanks for your help. You are so nice!

I am interested in ERSPAN, so I want to ask a more question.

1. What does vlan 1 mean? The physical switch connected to my host is not set any vlan id, but it seems that it can allow packets from vlan 1 port-profile. The vm with vlan 1 port-profile can get the dhcp ip address.

Thanks,

Caixia

prvc · ‎05-19-2010

Hi Caixia,

Its nice to know that you were able to get ERSPAN working.

Now about VLAN1, for any switching interface of any switch, when you do not configure any VLANs explicitly it will be a part of VLAN 1. VLAN1 is kind of a default VLAN that all switching devices will be part of. If you have a trunk interface then by default VLAN will be the native VLAN. That is in that interface VLAN 1 will not be tagged using the VLAN information. So now in your case since you just made the port as a switching interface (using switchport mode access), and did not specify any VLAN, that port by default became part of VLAN 1. This is not specific to ERSPAN. This is the general behaviour of any switching interface. Hope that helps.

Regards

Praveen