cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
17062
Views
2
Helpful
5
Replies

packet drops in interface vethernet.

My VMs on multiple hosts have packet loss, slowness and failure of various applications.

to see a show interfaces: the average  Output Packet drops in most of the Veth is: 6 million

swiasusis-vsm# show interface vethernet 94
Vethernet94 is up
    Port description is asusisv-fs1, Network Adapter 1
    Hardware is Virtual, address is 0050.56b3.2b38
    Owner is VM "asusisv-fs1", adapter is Network Adapter 1
    Active on module 9
    VMware DVS port 719
    Port-Profile is pp_TPPY_G1
    Port mode is access
    5 minute input rate 844083 bytes/second, 283 packets/second
    5 minute output rate 300443 bytes/second, 910 packets/second
    Rx
    56884399 Input Packets 56692863 Unicast Packets
    0 Multicast Packets 191536 Broadcast Packets
    95491825263 Bytes
    Tx
    1627828889 Output Packets 1592848343 Unicast Packets
    5905610 Multicast Packets 29074936 Broadcast Packets 1529361515 Flood Packets
    128878367330 Bytes
    0 Input Packet Drops 6826118 Output Packet Drops

Can you help with this problem?

5 Replies 5

Robert Burns
Cisco Employee
Cisco Employee

Greetings Juan,

Sure we can help.  Packet Loss on an "Ethernet" interface is normal - assuming you're using vPC-HM.  For drops on vEth interfaces this could be a "real" issue.


We'll need much more information to assist.

1. Provide a full copy of your config (send via personal message if you do not wish to post it here)

2. What OS is the attached VM

3. What applications are running on the VMs

4. What network adapter type are you using on the VM (e1000, Flexible, VMXNet) - I highly suggest you use the VMXNet3 driver.  I've seen the e1000 cause issues like this.

5. Have you configured Jumbo frames or anything out of the ordinary on the VM's OS?

Can you clear the counters on this interface and then look at them after a 5min interval.  this will tell us how quickly the errors are incrementing.

Let me know these details and I can advise further.

Regards,

Robert

I have this same issue output drops on nearly all of the Veth interfaces with one of my customer's 1000v deployments, I have had customer validate that they are using the VMXNet3 driver. What other potential causes are there for these output drops. I can send  the config and they are running 4.0.4.SV1.3b and the guest OSs are mostly server 2008 and 2003.

Before you look at "why" is it dropping you should take a look at "what" it's dropping.   For this we can do a packet capture on the VEM and see what's being dropped.  You might get some clues from this.

Here's how you'd do it.

1. On your VEM - we need the LTL (Local Target Logic) # of your VM's interface. These values are unique to each VEM (whereas vEth #'s are unique to the entire DVS).  You can do this from the VEM CLI.

vemcmd show port

2.  Once you have the LTL of your VM (VM LTL's start at 49 & increment), we'll setup a vempkt capture with the following commands from the VEM CLI.

vempkt stop

vempkt clear

vempkt capture drop ltl x  (where X = your VM's LTL from step 1)

(wait for 15-30sec)

vempkt stop

vempkt display detail all (to view locally)

vempkt display detail all > /tmp/vempkt_capture_drop.cap   (to export to a file)

3.  If you view the capture from the console you'll see some details about the dropped packets such as the source LTL (which you can possibly trace back to the VM using Step 1), source/destination MAC address, VLAN etc. **Note LTL in the range of 30x are your uplinks - meaning this traffic is to/from a host outside this particular VEM - In this case, trace the MAC address to the source/destination.) Unfortunately this is the only way to view the capture - unless you send the .cap file to me and I can run it through a parser which will generate a wireshark-readable file (courtesy of Michael Petrinovic @ TAC).  Once I parse the capture you can see if it's a particular protocol, destination etc...

You should look for patterns in the dropped packets.  I've seen strange multicast applications send out funny traffic which the VEM will end up dropping.  We'll need to see the packet contents to confirm.

Regards,

Robert

Hello Roberbur,

the parser is public available? where can I download it? If not, could I send you a .cap file to get it translate it to wireshark? I found it very useful this way to capture traffic on nexus1000v.

thanks in advance,

Fernando

From version 1.5 onward there is a parser is built into the system.  You can save a vempkt capture as a wireshark pcap you can offload and review.

vempkt pcap export <filename>

Then copy the pcap file off and open with wireshark.

Robert