1 2 Previous Next 15 Replies Latest reply: May 12, 2010 1:50 PM by bbaughn1008 RSS

Appropriate way to pin to manual sub-group with mac-pinning?

ryan.lambert

Hi folks,

 

Working through some specifics on our new 1KV deployment still, and I'm at the point where I have the following scenario:

 

Two iSCSI VLANs -

93 & 103

 

Each of these has a single port-profile that my 3 host servers use to create their VMK interfaces. I want to static pin one VLAN to a link, and have the second traverse the second uplink, so I effectively load-balance my iSCSI traffic.

 

I have 3 port-channels. 1 port-channel per server, 2 member links each:

 

Port-channel1 = server 1, Gig3/5, Gig3/8

Port-channel2 = server 2, Gig4/5, Gig4/8

Port-channel3 = server 3, Gig5/5, Gig5/8

 

My question is, can I use sub-group 0 & 1 for each pair of uplinks and specify "pinning id 0" and "pinning id 1" under my common vethernet port-profiles, or do I need to have sub-group 0/1 on po1, sub-group 2/3 on po2, and sub-group 4/5 on po3, and then create separate iSCSI port-profiles for each server that has the correct sub-group pinning ids?

 

Hope that makes sense. Made my head hurt typing it... just trying to get some clarification on the right way to do this.

 

Thanks!

  • 1. Re: Appropriate way to pin to manual sub-group with mac-pinning?
    ryan.lambert

    As an aside here are my port-profiles:

     

    port-profile type vethernet iSCSI-VLAN93

    vmware port-group

    switchport mode access

    switchport access vlan 93

    capability iscsi-multipath

    no shutdown

    system vlan 93

    state enabled

     

    port-profile type vethernet iSCSI-VLAN103

    vmware port-group

    switchport mode access

    switchport access vlan 103

    capability iscsi-multipath

    no shutdown

    system vlan 103

    state enabled

  • 2. Re: Appropriate way to pin to manual sub-group with mac-pinning?
    estine

    Hey Ryan -

     

    You can use subgroup 0/1 for each pair of uplinks and then use static pinning to pin the traffic to either subgroup.  Static pinning can be configured on the vethernet port profile or under the vethernet interface config mode.  The command for this is "pinning id <subgroupID>"

     

    Hope that helps.

     

     

    Thanks,

    Liz

  • 3. Re: Appropriate way to pin to manual sub-group with mac-pinning?
    nenduri

    Hi Ryan,

     

    The pinning of iscsi multipath capable ports is completely orthogonal to vPC-HM pinning. vPC-HM static pinning ("pinning id <sgid>") will not take effect for iscsi multipath ports.

     

    iscsi multipath capable ports (vmknics) get automatically pinnned to one of the Ethernet ports in round robin. User cannot configure this pinning from CLI.

     

    You can check the pinning information by the following command:

     

    vemcmd show iscsi pinning

     

    You can run the same command from VSM as below:

     

    module vem <mod#> exec  vemcmd show iscsi pinning

     

    Since vlan 93 and 103 carried on both uplinks, one get pinned to left and another to right.

     

    HTH.

     

    -Naren

  • 4. Re: Appropriate way to pin to manual sub-group with mac-pinning?
    ryan.lambert

    Hi Naren,

     

    I actually saw them getting pinned to the same uplinks (saw dynamic macs upstream on same physical switch port for both VLANs).

     

    Here's my output which seems to confirm that:

     

    N1KV-VSM1# module vem 4 exec vemcmd show iscsi pinning
      LTL   Device    IfIndex  Pinned_Uplink
       50   vmk1      1b030020        20
       51   vmk2      1b030030        20

     

    Any thoughts? If I can't manually pin them and I end up with this, not sure of how to evenly distribute my iSCSI traffic.

     

    VMK2 = 93

    VMK1 = 103

     

    Thanks!

  • 5. Re: Appropriate way to pin to manual sub-group with mac-pinning?
    nenduri

    Hi Ryan,

     

    Can you send the output of the following commands?

     

    vemcmd show trunk

    vemcmd show port

     

    vemcmd show bd 93

    vemcmd show l2 93

    vemcmd show dr 93

     

    vemcmd show bd 103

    vemcmd show l2 103

    vemcmd show dr 103

     

    thanks,

    Naren

  • 6. Re: Appropriate way to pin to manual sub-group with mac-pinning?
    ryan.lambert

    Hi Naren,

     

    We're in the process of rebuilding some servers, since this was a pre-deployment testing of hardware/software. Once we get the fresh hosts in place and attached to the DVS, I will shoot this info over to you. I should be able to replicate this pretty easily.    

     

    Should be in the next day or so, I hope.   

  • 7. Re: Appropriate way to pin to manual sub-group with mac-pinning?
    ryan.lambert

    Hi Naren!

     

    We see same behavior now that new VSM/VEMs are installed, so it did replicate itself.

     

    N1KV-VSM1# module vem 4 execute vemcmd show iscsi pinning
      LTL   Device    IfIndex  Pinned_Uplink
       48   vmk0      1b030000        20
       49   vmk1      1b030010        20

     

    Attached the requested output for you (vemcmds.txt)

     

    -Ryan

  • 8. Re: Appropriate way to pin to manual sub-group with mac-pinning?
    ryan.lambert

    Actually, this is confusing.

     

    Here's a sh mac-address table dyn off my physical switch interfaces (this is different than before):

     

    dc3# sh mac-address-table dynamic interface eth1/2

    VLAN      MAC Address       Type    Age       Port                         

    ---------+-----------------+-------+---------+------------------------------

    92        0050.567e.9488    dynamic 90        Eth1/2                       

    93        0050.5675.61d1    dynamic 190       Eth1/2                       

    95        0002.3d40.5b02    dynamic 0         Eth1/2    

     

    dc4# show mac-address-table dynamic interface eth1/2

    VLAN      MAC Address       Type    Age       Port                         

    ---------+-----------------+-------+---------+------------------------------

    91        0050.564f.aef1    dynamic 0         Eth1/2                       

    96        0002.3d20.5b02    dynamic 0         Eth1/2                       

    96        0050.567c.6d4b    dynamic 20        Eth1/2                       

    103       0050.5676.aec0    dynamic 100       Eth1/2           

     

     

     

     

    But... a few minutes later (I didn't drop link or anything on my NIC connected to dc3):

     

    dc4# sh mac-address-table dynamic interface eth1/2
    VLAN      MAC Address       Type    Age       Port                         
    ---------+-----------------+-------+---------+------------------------------
    91        0050.564f.aef1    dynamic 10        Eth1/2                       
    93        0050.5675.61d1    dynamic 80        Eth1/2                       
    96        0002.3d20.5b02    dynamic 10        Eth1/2                       
    103       0050.5676.aec0    dynamic 10        Eth1/2            

  • 9. Re: Appropriate way to pin to manual sub-group with mac-pinning?
    ryan.lambert

    Opened a TAC case on this as well, and waiting to talk to the Engineer, but still haven't had any luck on this.

     

    Hopefully this is just a matter of me doing something incorrectly...

  • 10. Re: Appropriate way to pin to manual sub-group with mac-pinning?
    ryan.lambert

    Well... good news and bad news:

     

    Bad news is according to my TAC engineer and the developer he was working with in the background, this is a bug in the current version SV1(2) and pertains only to iSCSI multipath configured port-profiles. They do not properly pin the VMKs to separate uplinks. I don't have a clean workaround for this in my environment, so I am stuck with all of my iSCSI VMKs traversing a single uplink.

     

    Good news is that it is supposed to be fixed in the next release, tentatively later this month.

     

    I didn't get a bug ID for it, but he may shoot it over in his closing notes.  

  • 11. Re: Appropriate way to pin to manual sub-group with mac-pinning?
    sekrishn

    Hi Ryan,

     

    Looking at the config, the two vmknics are in two different vlans 93 and 103. The iSCSI multipath auto-pinning works on a per-vlan basis. What this means is that when we pin vmk1 on vlan93, we scan the uplinks and pick the least loaded uplink, say Uplink1. When we pin vmk2, its on a different vlan 103 and the scan may pick Uplink1 again, since it is the least loaded on vlan103.

     

    N1K iSCSI multipathing assumes that the multiple paths to the iSCSI targets are on the same subnet, IOW same vlan. Hence we run the uplink selection on a per-vlan basis. If you move both the vmknics to the same vlan, then you should see they getting pinned to different uplinks.

     

    I also updated the TAC engineer, we can discuss the requirements for the paths on two different vlans.

     

    thx,

    - senthil

  • 12. Re: Appropriate way to pin to manual sub-group with mac-pinning?
    ryan.lambert

    Hi Senthil,

     

    Thanks for the information.

     

    We were/are (we just can't sustain that configuration long term) going to drop all of these back onto a single VLAN (per request of the TAC engineer) just to confirm pinning behavior was working as expected when all iSCSI is on one VLAN.

     

    Unfortunately, due to the issue in this blog below, we are forced into a 2 VLAN setup:

     

    http://virtualgeek.typepad.com/virtual_geek/2009/08/important-note-for-all-emc-clariion-customers-using-iscsi-and-vsphere.html

  • 13. Re: Appropriate way to pin to manual sub-group with mac-pinning?
    ryan.lambert

    Senthil,

     

    We've concluded (as expected) the VMKs do seem to pin properly if within the same VLAN.

     

    Of course, referring to my above post, I suppose I'm caught between a pair of stop signs at this point. EMC makes me do it one way, 1KV wants me to do it another way.

  • 14. Re: Appropriate way to pin to manual sub-group with mac-pinning?
    bbaughn1008

    We have the same issue.  We have a Netapp that we're connecting to over iSCSI, and to the EMC directive, the recommended design is 2 IPs on 2 VLANs, so we are stuck apparently unable to pin to different uplinks.  Any recommendations how to resolve this??

1 2 Previous Next