DPDK is the Data Plane Development Kit that consists of libraries to accelerate packet processing workloads running on a wide variety of CPU architectures. It is designed to run on x86, POWER, and ARM processors and licensed under the Open-Source BSD License.

Pktgen-DPDK is a traffic generator powered by DPDK.

This guide is written for application architects and build engineers who want to know how to build DPDK and set-up the test environment with Ampere^® Altra^® processors.

The configuration of the hardware is illustrated below.

Packet Generator:

• Ampere Altra Q80-30
• Mellanox ConnectX-6 Dx
• CentOS 8.5
• DPDK 23.03+ Pktgen-DPDK master

DUT:

• Ampere Altra Max M128-30
• Mellanox ConnectX-5 Ex
• CentOS 8.5
• DPDK 23.03

The packet generator and the DUT are connected back-to-back using a 100G cable.

Download the MLNX_OFED for Mellanox’s NIC from Mellanox's website[1] for the target OS. Then, install the MLNX_OFED using this command. It will prompt you if some packages are missing from the system. A reboot may be required.

Copy
./mlnxofedinstall --add-kernel-support --upstream-libs --dpdk --skip-repo

Install other packages needed for building DPDK.

Copy
sudo yum install python3 meson ninja-build python3-pyelftools numactl-devel -y 

Install packages needed for building Pktgen-DPDK.

Copy
sudo yum install  libpcap-devel -y 

Download the DPDK source code from DPDK website: https://fast.dpdk.org/rel/ and then build using the following commands.

Copy
wget https://fast.dpdk.org/rel/dpdk-23.03.tar.gz 
tar zxf dpdk-23.03.tar.gz 
cd dpdk-23.03 
meson build 
ninja -C build 
sudo ninja -C build install 

The DPDK libraries and binaries will be installed at /usr/local/lib64 and /usr/local/bin/.

This section is only applied on the packet generator system. Download the Pktgen-DPDK source code from github. As Pktgen-DPDK uses the DPDK libraries installed in prior section, please select a version which is close to the DPDK version. Then build using the following commands:

Copy
export PKG_CONFIG_PATH=/usr/local/lib64/pkgconfig 
git clone https://github.com/pktgen/Pktgen-DPDK.git
cd Pktgen-DPDK 
make

Before starting your testing, some settings should be applied on both the packet generator and DUT system. Some options may be different depending on the Operating System (OS).

Start the pktgen application on the system used as packet generator. In this example, we use one 100G port as pkt transmitter.

Copy
export PKG_CONFIG_PATH=/usr/local/lib64/pkgconfig 
cd Pktgen-DPDK 
./Builddir/app/pktgen --file-prefix pktgen -a 0000:01:00.0 -l 40-59 -n 8 -- -N -T -m [42-43:44-59].0

Enable the range from the prompt of Pktgen.

Copy
Pktgen:/> set 0 dst mac <mac> 
Pktgen:/> range 0 dst mac <mac> 00:00:00:00:00:00 00:00:00:00:00:00 00:00:00:00:00:00 
Pktgen:/> range 0 size 64 64 64 0 
Pktgen:/> enable 0 range 
Pktgen:/> start 0

Below is the screenshot of sending packets at 148.8Mpps, which is the line rate of 100G NIC interface.

Table-1 lists the line rate of different ethernet link speeds with different packet sizes.

Start the dpdk-testpmd application on the system under test. Here is an example (CMD-1) with 1 core and 3 rx queue/3 tx queue.

Copy
(CMD-1) dpdk-testpmd --file-prefix testpmd -l 40-79 -n 8 -a 0000:01:00.0,mprq_en=1,rxqs_min_mprq=1,rxq_cqe_comp_en=1,rxq_pkt_pad_en=1,txq_inline_max=1024,txq_inline_mpw=1024,txqs_min_inline=1,txq_mpw_en=1 -- -i --rss-ip --nb-cores=1  --burst=128 --txq=3 --rxq=3  --mbcache=512 --rxd=1024 --txd=1024  -a

In this example, we use the data inline mode by enabling the Multi-Packet Rx Queue (MPRQ), Enhanced Multi-Packet Write (eMPW) and CQE compression of Mellanox ConnectX-5, ConnectX-6 NIC. These features can save the PCIe bandwidth and improve performance when the PCIe back pressure is high. However, it consumes additional CPU cycles for data copying. It may hurt performance when PCIe back pressure is not high. Here is another example (CMD-2) which disables the data inline mode:

Copy
(CMD-2) dpdk-testpmd --file-prefix testpmd -l 40-79 -n 8 -a 0000:01:00.0,mprq_en=0,rxqs_min_mprq=1,rxq_cqe_comp_en=1,rxq_pkt_pad_en=1,txq_inline_max=0,txq_inline_mpw=0,txqs_min_inline=1,txq_mpw_en=1 -- -i --rss-ip --nb-cores=1  --burst=128 --txq=3 --rxq=3  --mbcache=512 --rxd=1024 --txd=1024  -a

The Rx/Tx comparison of single core between CMD-1(inline mode) and CMD-2(non-inline mode) is as below:

When the PPS is not high, CMD-2 is recommended and when it reaches to high PPS, CMD-1 is recommended.

Some useful commands of testpmd:

show port stats all – Display the statistics data of each port

show fwd stats all – Display the forward statistics of each port

clear port stats all – Clear the statistics data of each port

You can increase the core count and txq/rxq number to achieve the line rate. Below is the screenshot indicating that both Rx-pps and Tx-pps are 148.8Mpps.

For more details about the dpdk-testpmd command-line options, please refer to Testpmd Application User Guide.

For more details about the poll mode driver(pmd) of Mellanox’s CX5 and CX6, please refer to “MLX5 poll mode driver”.

1. https://network.nvidia.com/products/infiniband-drivers/linux/mlnx_ofed
2. https://www.kernel.org/doc/html/latest/arm64/hugetlbpage.html
3. https://doc.dpdk.org/guides/testpmd_app_ug/
4. https://doc.dpdk.org/guides-19.11/nics/mlx5.html
5. https://pktgen-dpdk.readthedocs.io/en/latest/