DPDK Cryptography Build and Tuning Guide
Build and Run DPDK with Cryptography PMD
One of the many use cases customers run on Ampere powered systems is packet processing workloads built on DPDK. Ampere has published a Setup and tuning guide for DPDK to assist customers with getting the best performance from these workloads. Since many customers make heavy use of encryption/decryption operations in their DPDK applications we are supplementing the existing DPDK tuning guide with additional information on crypto library support and how to build DPDK with these crypto libraries.
These steps should happen before building DPDK library.
ARMv8 Crypto Driver
The ARMv8 crypto poll mode driver enables use of crypto extensions to ARMv8 that optimize chained operations. The core functions of this driver are written in assembly. It is published by Arm at https://github.com/ARM-software/AArch64cryptolib.git.
ARMv8 Crypto PMD supports the following algorithm pairs:
Cipher algorithms:
- RTE_CRYPTO_CIPHER_AES_CBC
Authentication algorithms:
- RTE_CRYPTO_AUTH_SHA1_HMAC
- RTE_CRYPTO_AUTH_SHA256_HMAC
Build DPDK with ARMv8 crypto PMD:
Download and build AArch64 crypto library source code (Assumes current directory is /home/ampere/):
git clone https://github.com/ARM-software/AArch64cryptolib.git
cd AArch64cryptolib
make OPT=big EXTRA_CFLAGS="-march=armv8.2-a+crypto"
sudo echo “/home/ampere/AArch64cryptolib” > /etc/ld.so.conf.d/armcrypto.conf
sudo ldconfig Reference: https://doc.dpdk.org/guides/cryptodevs/armv8.html
OpenSSL Crypto Driver
For best performance with OpenSSL drivers use either OpenSSL 3.2 or 1.1.1. Based on the results of our testing these versions provide the best performance, and versions 3.0.x and 3.1.x should be avoided due to significantly worse results.
OpenSSL Crypto PMD supports the following algorithm pairs:
Cipher algorithms:
- RTE_CRYPTO_CIPHER_3DES_CBC
- RTE_CRYPTO_CIPHER_AES_CBC
- RTE_CRYPTO_CIPHER_AES_CTR
- RTE_CRYPTO_CIPHER_3DES_CTR
- RTE_CRYPTO_CIPHER_DES_DOCSISBPI
Authentication algorithms:
- RTE_CRYPTO_AUTH_AES_GMAC
- RTE_CRYPTO_AUTH_MD5
- RTE_CRYPTO_AUTH_SHA1
- RTE_CRYPTO_AUTH_SHA224
- RTE_CRYPTO_AUTH_SHA256
- RTE_CRYPTO_AUTH_SHA384
- RTE_CRYPTO_AUTH_SHA512
- RTE_CRYPTO_AUTH_MD5_HMAC
- RTE_CRYPTO_AUTH_SHA1_HMAC
- RTE_CRYPTO_AUTH_SHA224_HMAC
- RTE_CRYPTO_AUTH_SHA256_HMAC
- RTE_CRYPTO_AUTH_SHA384_HMAC
- RTE_CRYPTO_AUTH_SHA512_HMAC
AEAD algorithms:
- RTE_CRYPTO_AEAD_AES_GCM
- RTE_CRYPTO_AEAD_AES_CCM
Asymmetric Crypto algorithms:
- RTE_CRYPTO_ASYM_XFORM_RSA
- RTE_CRYPTO_ASYM_XFORM_DSA
- RTE_CRYPTO_ASYM_XFORM_DH
- RTE_CRYPTO_ASYM_XFORM_MODINV
- RTE_CRYPTO_ASYM_XFORM_MODEX
Download and Install OpenSSL 3.2.0:
The OpenSSL libraries along with each OS distribution are quite different.
That will make performance variance on different OS distributions.
In order to keep the performance consistent, please download and install the OpenSSL 3.2.0.
wget https://github.com/openssl/openssl/archive/refs/tags/openssl-3.2.0.tar.gz
tar zxf openssl-3.2.0.tar.gz
cd openssl-openssl-3.2.0
./Configure -mcpu= neoverse-n1
make -j`nproc`
sudo make -j `nproc` install
sudo echo “/usr/local/lib” > /etc/ld.so.conf.d/openssl.conf
sudo ldconfig Reference: https://doc.dpdk.org/guides/cryptodevs/openssl.html
IPSec Multi-buffer library for Aarch64
IPSec Multi-buffer library for Aarch64 supports following algorithm pairs:
Cipher algorithm:
- SNOW3G-UEA2
- ZUC-EEA3
- ZUC-EEA3-256
Authentication algorithm:
- SNOW3G-UIA2
- ZUC-EIA3
- ZUC-EIA3-256
Download and build ipsec-mb library:
git clone https://gitlab.arm.com/arm-reference-solutions/ipsec-mb
cd ipsec-mb
make
make install PREFIX=/usr/local/ Reference: https://doc.dpdk.org/guides/cryptodevs/snow3g.html
Build DPDK with Crypto Support
On CentOS
export LD_LIBRARY_PATH=/home/ampere/AArch64cryptolib:/usr/local/lib:/lib64
export PKG_CONFIG_PATH=/home/ampere/AArch64cryptolib/pkgconfig:/usr/local/lib/pkgconfig:/lib64/pkgconfig On Ubuntu
export LD_LIBRARY_PATH=/home/ampere/AArch64cryptolib:/usr/local/lib:/usr/local/lib/aarch64-linux-gnu:/lib/aarch64-linux-gnu
export PKG_CONFIG_PATH=/home/ampere/AArch64cryptolib/pkgconfig:/usr/local/lib/pkgconfig:/usr/local/lib/aarch64-linux-gnu/pkgconfig:/lib/aarch64-linux-gnu/pkgconfig Build DPDK
wget https://fast.dpdk.org/rel/dpdk-23.07.tar.gz
tar zxf dpdk-23.07.tar.gz
cd dpdk-23.07
meson build
ninja -C build
ninja -C build install Check the config of supported crypto device for armv8, openssl, ipsec_mb:
- armv8, bcmfs, caam_jr, ccp, cnxk, dpaa_sec, dpaa2_sec, ipsec_mb,mlx5, nitrox, null, octeontx, openssl, scheduler, virtio,
Crypto Performance Test
The following performance test was performed on Ampere Altra Q80-30. The performance data will be different if different SKU is used. Please refer to the later section “Tunning Guide” for hardware, BIOS, OS settings before performance testing.
Test AES-CBC-128/SHA1-HMAC Performance with single core using crypto_armv8:
sudo usertools/dpdk-hugepages.sh –setup 2G
cd build/app
./dpdk-test-crypto-perf --socket-mem 2048,0 --legacy-mem --vdev crypto_armv8 -l 9,10 -n 8 -- --buffer-sz 64,128,256,512,1024,2048 --optype cipher-then-auth --ptest throughput --auth-key-sz 64 --cipher-key-sz 16 --devtype crypto_armv8 --cipher-iv-sz 16 --auth-op generate --burst-sz 32 --total-ops 10000000 --silent --digest-sz 12 --auth-algo sha1-hmac --cipher-algo aes-cbc --cipher-op encrypt
lcore id Buf Size Burst Size Enqueued Dequeued Failed Enq Failed Deq MOps Gbps Cycles/Buf
10 64 32 10000000 10000000 0 0 6.4872 3.3214 3.85
10 128 32 10000000 10000000 0 0 4.6440 4.7555 5.38
10 256 32 10000000 10000000 0 0 2.9792 6.1014 8.39
10 512 32 10000000 10000000 0 0 1.7663 7.2348 14.15
10 1024 32 10000000 10000000 0 0 0.9745 7.9831 25.65
10 2048 32 10000000 10000000 0 0 0.5136 8.4144 48.68 Test AES-CBC-128/SHA2-256-HMAC Performance with single core using crypto_armv8:
cd build/app
./dpdk-test-crypto-perf --socket-mem 2048,0 --legacy-mem --vdev crypto_armv8 -l 9,10 -n 8 -- --buffer-sz 64,128,256,512,1024,2048 --optype cipher-then-auth --ptest throughput --auth-key-sz 64 --cipher-key-sz 16 --devtype crypto_armv8 --cipher-iv-sz 16 --auth-op generate --burst-sz 32 --total-ops 10000000 --silent --digest-sz 12 --auth-algo sha2-256-hmac --cipher-algo aes-cbc --cipher-op encrypt
lcore id Buf Size Burst Size Enqueued Dequeued Failed Enq Failed Deq MOps Gbps Cycles/Buf
10 64 32 10000000 10000000 0 0 6.7274 3.4444 3.72
10 128 32 10000000 10000000 0 0 4.8872 5.0045 5.12
10 256 32 10000000 10000000 0 0 3.1074 6.3639 8.05
10 512 32 10000000 10000000 0 0 1.8370 7.5244 13.61
10 1024 32 10000000 10000000 0 0 1.0115 8.2860 24.72
10 2048 32 10000000 10000000 0 0 0.5326 8.7260 46.94 Test AES-GCM-128 Performance with single core using crypto_openssl:
cd build/app
./dpdk-test-crypto-perf --socket-mem 2048,0 --legacy-mem --vdev crypto_openssl -l 9,10 -n 8 -- --aead-key-sz 16 --buffer-sz 64,128,256,512,1024,2048 --optype aead --ptest throughput --aead-aad-sz 16 --devtype crypto_openssl --aead-op encrypt --burst-sz 32 --total-ops 10000000 --silent --digest-sz 16 --aead-algo aes-gcm --aead-iv-sz 12
lcore id Buf Size Burst Size Enqueued Dequeued Failed Enq Failed Deq MOps Gbps Cycles/Buf
10 64 32 10000000 10000000 0 0 4.6605 2.3862 5.36
10 128 32 10000000 10000000 0 0 4.2379 4.3396 5.90
10 256 32 10000000 10000000 0 0 3.4681 7.1027 7.21
10 512 32 10000000 10000000 0 0 2.6914 11.0240 9.29
10 1024 32 10000000 10000000 0 0 1.8423 15.0920 13.57
10 2048 32 10000000 10000000 0 0 1.1341 18.5805 22.04 Test snow3g-uea2 cipher-only with single core using crypto_snow3g:
cd build/app
./dpdk-test-crypto-perf --socket-mem 2048,0 --legacy-mem --vdev crypto_snow3g -l 9,10 -n 8 -- --devtype crypto_snow3g --ptest throughput --pool-sz 16384 --total-ops 10000000 --burst-sz 32 --optype cipher-only --cipher-algo snow3g-uea2 --cipher-iv-sz 16 --auth-op generate --cipher-key-sz 16 --buffer-sz 64,128,256,512,1024,2048 --cipher-op encrypt
lcore id Buf Size Burst Size Enqueued Dequeued Failed Enq Failed Deq MOps Gbps Cycles/Buf
10 64 32 10000000 10000000 0 0 3.4215 1.7518 7.31
10 128 32 10000000 10000000 0 0 2.6380 2.7013 9.48
10 256 32 10000000 10000000 0 0 1.8216 3.7306 13.72
10 512 32 10000000 10000000 0 0 1.1236 4.6022 22.25
10 1024 32 10000000 10000000 0 0 0.6368 5.2166 39.26
10 2048 32 10000000 10000000 0 0 0.3411 5.5878 73.30 Performance Scaling with Core Counts
The crypto throughput on Ampere Altra is linear with the core count. Here is the AES-GCM-128 throughput with Buffer size=1024 at different core counts:
| Core Count | Throughput(Gbps) |
|---|---|
| 1 | 15.12 |
| 2 | 30.24 |
| 4 | 59.67 |
| 8 | 121.63 |
| 16 | 238.63 |
And snow3g-uea2 cipher-only throughput with Buffer size=1024 at different core counts:
| Core Count | Throughput(Gbps) |
|---|---|
| 1 | 5.2125 |
| 2 | 10.4234 |
| 4 | 20.8493 |
| 8 | 41.6977 |
| 16 | 83.3840 |
Run l2fwd with Crypto
DPDK provides an example application l2fwd-crypto which can do L2 forwarding with crypto. To perform this test, please follow the DPDK-setup-and-tuning-guide and setup the Pktgen-dpdk as a packet generator.
Forwarding with AES-GCM-128bit crypto, 1 port, 1 core, pktsize=1024B:
./build/l2fwd-crypto -l 10-15 -n 8 -a 0000:01:00.0 --vdev crypto_openssl -- -p 0x1 --chain AEAD --aead_op ENCRYPT --aead_algo aes-gcm -T 1
Statistics for port 0 ------------------------------
Packets sent: 1339751
Packets received: 1339780
Packets dropped: 0
Crypto statistics ==================================
Statistics for cryptodev 0 -------------------------
Packets enqueued: 1339780
Packets dequeued: 1339751
Packets errors: 0 Forwarding with AES-CBC/SHA1-HMAC crypto, 1 port, 1 core, pktsize=1024B:
./build/l2fwd-crypto -l 10-15 -n 8 -a 0000:01:00.0 --vdev crypto_armv8 -- -p 0x1 --chain CIPHER_HASH --cipher_op ENCRYPT --cipher_algo aes-cbc --cipher_key 00:01:02:03:04:05:06:07:08:09:0a:0b:0c:0d:0e:0f --auth_op GENERATE --auth_algo sha1-hmac --auth_key 10:11:12:13:14:15:16:17:18:19:1a:1b:1c:1d:1e:1f -T 1
Statistics for port 0 ------------------------------
Packets sent: 869828
Packets received: 869856
Packets dropped: 0
Crypto statistics ==================================
Statistics for cryptodev 0 -------------------------
Packets enqueued: 869856
Packets dequeued: 869828
Packets errors: 0 Tunning Guide
Hardware Configure
- 1 DIMM Per Channel memory population, 3200 MT/s;
BIOS Settings
- Advanced->ACPI Settings->Enable CPPC [Disabled]
- Advanced->ACPI Settings->Enable LPI [Disabled]
- Chipset->CPU Configuration->ANC mode [Monolithic]
- Chipset->CPU Configuration-> SLC Replacement Policy [Enhanced Least Recently Used]
- Chipset->CPU Configuration->L1/L2 Prefetch [Enabled]
- Chipset->CPU Configuration->SLC as L3$ [Disabled]
OS Settings
- Use gcc with version > 10 and recommended build options
- Reference: https://amperecomputing.com/tutorials/gcc-guide-ampere-processors
- Set Hugepage. Example on CentOS with 64k kernel page:
- echo 100 > /sys/devices/system/node/node0/hugepages/hugepages-524288kB/nr_hugepages
Library Version Selection
- Checkout the latest library code for AArch64cryptolib, ipsec-mb;
- Use OpenSSL with version 3.2.0 or 1.1.1
Reference
- https://amperecomputing.com/tuning-guides/DPDK-setup-and-tuning-guide
- https://amperecomputing.com/tutorials/gcc-guide-ampere-processors
- https://doc.dpdk.org/guides/cryptodevs/armv8.html
- https://doc.dpdk.org/guides/cryptodevs/openssl.html
- https://doc.dpdk.org/guides/cryptodevs/snow3g.html
- https://doc.dpdk.org/guides-16.04/sample_app_ug/l2_forward_crypto.html
Ampere Computing LLC
4655 Great America Parkway Suite 601
Santa Clara, CA 95054
