cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
3547
Views
5
Helpful
0
Comments
amichayp
Level 1
Level 1

Mellanox develops the IDG4400 Flex, a high-availability, fully-programmable, Ethernet platform based on its Indigo NPS-400 network processor capable of sustaining high-speed packet-processing for Layer 2–7 applications.

The abilities and scale of this product posed a challenge from a testing point of view. The requirements for the testing included:

  • Stateful Flow Table (SFT) – up to 550Gb/s 200M flows.
  • Deep Packet Inspection (DPI) Performance – up to 400Gb/s, 3M CPS and 100M flows.
  • In order to test the stateful software component (SFT) and the SFT client – DPI (a stateful service which uses the SFT as infrastructure) – we needed a stateful traffic generator with a realistic traffic mix.

capture.pngHigh Level Building Blocks for L4-7 Applications

As a former leader of the Openstack QA Activities, I started diving into a new Open Source Project to find a solution (traffic generator) that would answer my testing environment requirements.    

Why TRex?

1) High performance – 150Gb with Mellanox NICs. (See Mellanox ConnectX-4/5 support section in TRex Stateful Manual document)

2) In large bandwidth scenarios, the stateful distribution of packets in TRex is much more realistic than competing technologies (i.e. there are no flow bursts).

3) X millions of flows.

4) Easy installation with clear and simple documentation.

5) Cost effective use of existing servers and NICs.

Setup Topology & TRex Server Details:

topology-setup.PNG

ComponentsDetails
Server TypeHP ProLiant DL380 Gen9 / Dell PowerEdge R730
DistributionRed Hat 7.2
HCA / NIC Type

Mellanox ConnectX-4 / ConnectX-5

MCX416A-CCAT / MCX516A-CCAT

PCIe

2x PCIe16 + 2x PCIe8

Driver Version3.4-2.0.0.0 / 4.0-1.0.1.0

TRex Configuration File:

Capture1.JPG

Partial Mellanox Application Mix YAML Configuration:

Capture.JPG

The Mellanox Traffic Mix represents Mellanox' s view of the traffic in relevant locations in the network. It has been verified by key vendors and is tuned toward traffic in the data center.

Mellanox is using this traffic mix when profiling and optimizing its stateful L4-7 technologies. Although different, both the SFR traffic profile and the Mellanox traffic profile are attempts at depicting average stateful traffic.

A key differentiator between the profiles is the number of concurrent flows (i.e. active or alive flows) for 1Gb/sec.

The SFR traffic profiles have ~5K active flows for 1Gbs, while in our case Mellanox assumes more than ~50K flows per 1Gbs of traffic, Of course this traffic assumption affects the overall traffic that the TRex can produce. This requirement also imposed a performance degradation on TRex as we had a total of 6M active flows for total of 150Gb/sec. traffic (without which we could have reached total of ~300Gb/sec with one server), see here for more information

https://trex-tgn.cisco.com/trex/doc/trex_manual.html#_more_active_flows

This profile exercises the DUT performance too, and we expect that a similar feature running on x86 will experience a performance degradation with our Mellanox traffic profile.

https://github.com/cisco-system-traffic-generator/trex-profiles

We could have improved TRex performance by creating a distribution of flows over time as described in the manual, but we chose not to do so.

Snapshot of TRex run Console:

Capture3.JPG

TRex Missing Functionality:

The following items are lacking from TRex in our view:

1) TCP Stack

2) Simulation of packet loss and simulation of retransmission handling.

3) Routing Emulation Support - BGP/OSPF/ARP/VRRP/DHCP/PIM

4) Stateless GUI Gaps:

  • Build/send L2/L3 frames of control protocols, such as STP/LACP/OSPF/BGP.
  • L2 protocol emulation (LACP, STP and IGMP) support.
  • Incremental support on the following fields - VLAN/PRI/TCP Port Number/MTU.
  • Capture capabilities

At this stage we spreading our use of TRex stateful/stateless by sharing the knowledge and resources between teams in order to achieve more from this unique tool from a performance/scale/automation perspective.


I would like to thank our DPDK Team (Olga and Shahaf) for integrating TRex in Mellanox and, of course,  Hanoch and his team for continually providing quick support.

Thanks,

Amichay.

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: