RDMA Mellanox support for Cisco UCS.

[Kristofer Kawas]

Hi

I wonder if anyone can help or answer me if there is support from RDMA Mellanox and Cisco UCS B series or fabric interconnect. It is possible to connect it technically.

I have customers who have Cisco UCS B Series more Windows 2012 R2 HyperV installed, who now want to connect RDMA Mellanox storage support

[Walter Dey]

Cisco UCS Rack Servers with Mellanox ConnectX EN RoE

https://www.cisco.com/web/strategy/docs/finance/mellanox_low_latency_wp.pdf

http://www.cisco.com/c/dam/en/us/products/collateral/switches/nexus-7000-series-switches/le_32804_pb_hpc10ge.pdf

One could connect a rack Mount Server with the proper VIC adaptor to a UCS Fabric interconnect, and have all the benefits of a B-series System (e.g. Service profiles)

[Kristofer Kawas]

Hi

Thanks for your answer,the problem is my customer wants to connect two Mellanox Ethernet switchs to a UCS blade systems by FI and 1340 mezzenin card, I wonder if it will work, these two switches will be used for SAN traffic that I mean that connects the SAN system to Mellanox switches and then to FI and on to all blades,

I have no rack servers all servers in the system are B200 and 5108 Chassis.

/Kristofer

[Walter Dey]

I don't think that Infiniband was ever asked for.

Can you please clarify what you mean by

...my customer wants to connect two Mellanox Ethernet switchs to a UCS blade systems by FI and 1340 mezzenin card, I wonder if it will work, these two switches will be used for SAN traffic that I mean that connects the SAN system to Mellanox switches and then to FI and on to all blades...

do you mean IP storage traffic, e.g. SMB (RoCE) ?

As was indicated by jjmcdani, VIC 1340 supports RoCE, and therefore it is not necessary to have any Mellanox adaptor on the blade.

jjmcdani

jjmcdani

[Kristofer Kawas]

Hi

Thanks walter.

What I meant was that I got a question from a customer who wants to change the usual FC storage with SMB RDMA and that they use two Mellanox switches, so I wondered if it is possible to connect the to the UCS system that they already has two pieces 5108 and 8 B200 blades.

And now i got the information about that and the conclusion that it is possible to do so with the right firmware version and the right mezzanine card.

[Walter Dey · Accepted solution]

Hi Kristofer

Important health warning: RoCE means RDMA over Converged Ethernet.

If you watch the YouTube mentioned above, you will see, that the SMB Server as well as Client are on UCS blades; therefore no interop Situation; everything happens within a UCS Domain.

In your situation, you have a Extended Converged Ethernet Network (Cisco and Mellanox); your Clients might be on blades, the Servers attached to the Mellanox Switches.

This  is essentially a interop question; I have no clue if it has  been tested or not.

Walter.

[jjmcdani]

Yes, keep in mind this release is for RoCE v1, which is non-routable.

RoCE v2 is routable, support for that on the VIC/UCS is coming within a year I am told.

We need to find a CVD on this.  I would assume in this release we only support within the same UCS domain... might support direct attached to an array that supported it as well.  Will have to check.

[jjmcdani · Accepted solution]

Hello Kristofer.

A more important question might be "Why Infiniband?".  With UCS Manager 2.2(4b) firmware Cisco now supports Microsoft SMB Direct with RoCE (RDMA over Converged Ethernet).  Check the requirements and see if this is a better solution for you.  It will require the newer VIC1340 adapter in each blade requiring this feature:

UCSM 2.2 GUI Configuration Guide

Get the Most Complete Solution

You Tube Video from Cisco Joe