Ampere Computing Logo
Contact Sales
Ampere Computing Logo
Ampere Cloud Native Processor solutions

NGINX Tuning Guide

For Ampere Altra Processors

Overview
Build Prerequisites
Building and Installation
Compilation
NGINX Configuration
Tuning For Performance
Worker Processes
Worker Connections
Keepalive Connections
Sendfile
Appendix
References
Overview

NGINX is an open source, high performance HTTP and reverse proxy server with many other web service-related features bundled. It is also used as a load balancer in the cloud. NGINX implements an asynchronous event-driven architecture to handle incoming requests and is built to offer a low memory footprint and high concurrency. NGINX is the most popular web server among high-traffic websites as of 20211.

The purpose of this guide is to describe techniques to run NGINX in an optimal manner on Ampere® Altra® processors.

The NGINX workload is comprised of a target running the NGINX web server and client(s) running the Wrk HTTP traffic generation program.

Build Prerequisites

Running an application in a performant manner starts with building it correctly and using the appropriate compiler flags. When running on Ampere® Altra® processors, we recommend building from source with the GCC compiler version 10 or newer. Newer compilers tend to have better support for new processor features and incorporate more advanced code generation techniques.

We have used CentOS8 as the operating system for our testing.


  • Download and install GCC 10 from SCL repository:
sudo yum -y install yum install scl-utils scl-utils-build sudo yum -y install gcc-toolset-10-gcc scl enable gcc-toolset-10 bash

For other operating systems like Ubuntu 20.04 LTS and Debian, GCC 10.2.1 is available and can be installed directly from the respective repositories.

Building and Installation

NGINX can be installed from repositories that the OS package manager offers or can be built directly from source. A comprehensive NGINX installation guide can be found in the official documentation. We recommend installing from source for flexibility, control, and the ability to configure specific modules.

To build NGINX optimized for Ampere® Altra® processors family, additional compile flags which can leverage hardware features can be added during compilation stage. NGINX source code for compilation can be obtained from NGINX download page. The stable version NGINX1.20.0 is used in our example. Installation from the source requires certain libraries and additional modules that will be compiled into the binary.


  • Execute the following steps to install the dependencies.
yum install -y zlib-devel pcre-devel git wget yum -y groupinstall "Development Tools" # needed for openssl 3.x compilation yum install perl-IPC-Cmd

NGINX, at its core, is a collection of modules. One can pick and choose modules depending on the functionality needed.

  • To support https connections, download the latest code for these additional sources from the respective git repositories.
git clone https://github.com/openssl/openssl.git git clone https://github.com/simpl/ngx_devel_kit.git

  • Download NGINX source.
NGINX_VER=1.20.0 wget http://nginx.org/download/nginx-$NGINX_VER.tar.gz tar -xvf nginx-$NGINX_VER.tar.gz
Compilation

The configure options are specified with the ./configure script in the source root directory. It allows for configuration of various NGINX parameters including the paths to the sources, compiler options, and the list of modules. The script also creates a makefile needed for the next step - compilation.


  • Nonstandard compiler paths and additional compiler options can be set using the CC environmental variable.
# Note: set NGINX_DIR variable for the NGINX installation destination ./configure \ --with-http_ssl_module \ --with-http_stub_status_module \ --with-openssl=../openssl \ --with-http_v2_module \ --add-module=../ngx_devel_kit \ --prefix=$NGINX_DIR make -j`nproc` install
NGINX Configuration

In our example, NGINX is configured to support https and gzip compression. Please refer to the nginx.conf and gzip.conf files shown in the appendix to configure the server.


  • Create your own ssl keys using OpenSSL and copy it to the system under test.
sudo cp key/NGINX_TEST_SSL.crt /etc/pki/tls/certs/ sudo cp key/NGINX_TEST_SSL.key /etc/pki/tls/private/ sudo cp key/NGINX_TEST_SSL.csr /etc/pki/tls/private/

  • Start the server once NGINX is configured.
cp test.html $NGINX_DIR/html cp gzip.conf $NGINX_DIR/conf/ cp nginx.conf $NGINX_DIR/conf/nginx.conf #start the server sudo $NGINX_DIR/sbin/nginx
Tuning For Performance

There are hundreds of settings that can alter the functionality and performance of NGINX. What is listed below are just some of the more common knobs that can be used. The NGINX documentation is the recommended resource to understand all the settings.

Worker Processes

NGINX worker processes (worker_processes) handle incoming web server requests. It is commonly recommended to set this to auto to match the number of processes to the number of CPU cores that are available on the platform. With the Ampere Altra family of processors pushing the limits of core count, we recommend studying scaling of your workload by trying values lower than the number of CPU cores to find the ideal value for your usage.

Worker Connections

NGINX worker connections (worker_connections) represents the number of simultaneous connections that all available worker processes are able to handle. The default for this is 512 connections. Ampere Altra processor-based platforms have enough resources to support a larger number and it is recommended to find the right number for your usage. In our testing, we have used a value of 10240

Keepalive Connections

Keepalive connections can improve performance dramatically by reducing CPU and network overheads needed to manage connections. There are two settings related to keepalives.

keepalive_requests – this is the number of requests a client can make over a single keepalive connection. The default value is 100, but higher values can be used for testing. In our tests, we have used a high value of 100 billion.

keepalive_timeout – this denotes how long an idle keepalive connection remains open. Our tests use 300 seconds.

Sendfile

By default, NGINX handles file transmission directly by copying it into the buffer before sending it. The sendfile directive eliminates this step and enables a zero-copy by directly copying from one file descriptor to the other. It can be enabled by using the sendfile directive in the configuration file. It is important to note that turning sendfile on will enable data to bypass user space, so it is not subject to the regular NGINX processing chain and filter like gzip.

Appendix

nginx conf file

# nginx.conf # For more information on configuration, see: # * Official English Documentation: http://nginx.org/en/docs/ # * Official Russian Documentation: http://nginx.org/ru/docs/ user nginx; worker_processes auto; worker_rlimit_nofile 104857600; error_log /var/log/nginx/error.log; pid /run/nginx.pid; # Load dynamic modules. See /usr/share/doc/nginx/README.dynamic. include /usr/share/nginx/modules/*.conf; events { use epoll; accept_mutex off; worker_connections 10240; } http { log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; #access_log /var/log/nginx/access.log main; access_log off; open_file_cache max=10240000 inactive=60s; open_file_cache_valid 80s; open_file_cache_min_uses 1; keepalive_requests 100000000000; keepalive_timeout 300s; sendfile on; tcp_nopush on; tcp_nodelay on; types_hash_max_size 4096; include /etc/nginx/mime.types; default_type application/octet-stream; # Load modular configuration files from the /etc/nginx/conf.d directory. # See http://nginx.org/en/docs/ngx_core_module.html#include # for more information. include /etc/nginx/conf.d/*.conf; server { listen 80; listen [::]:80; server_name _; root /usr/share/nginx/html; # Load configuration files for the default server block. include /etc/nginx/default.d/*.conf; error_page 404 /404.html; location = /40x.html { } error_page 500 502 503 504 /50x.html; location = /50x.html { } } server { listen 443 ssl http2; listen [::]:443 ssl http2; server_name _; root /usr/share/nginx/html; ssl_certificate "/etc/pki/tls/certs/NGINX_TEST_SSL.crt"; ssl_certificate_key "/etc/pki/tls/private/NGINX_TEST_SSL.key"; ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; #ssl_ciphers ALL:!aNULL:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP; #ssl_ciphers "AES128+SHA256 NULL+SHA256 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS !MEDIUM !RC4"; ssl_ciphers "AES128+SHA256 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS !MEDIUM !RC4"; ssl_prefer_server_ciphers on; # Load configuration files for the default server block. include /etc/nginx/default.d/*.conf; error_page 404 /404.html; location = /40x.html { } error_page 500 502 503 504 /50x.html; location = /50x.html { } }

gzip conf file

gzip on; gzip_min_length 100; gzip_buffers 8 32k; gzip_types text/plain text/css application/x-javascript text/xml application/xml text/javascript; gzip_vary on;
References
Created At : June 1st 2023, 10:40:50 pm
Last Updated At : June 1st 2023, 10:40:54 pm
Ampere Logo

Ampere Computing

4655 Great America Parkway

Suite 601 Santa Clara, CA 95054

image
image
 |  |  |  |  |  | 
© 2022 Ampere Computing LLC. All rights reserved. Ampere, Altra and the A and Ampere logos are registered trademarks or trademarks of Ampere Computing.
This site is running on Ampere Altra Processors.