NGINX Tuning Guide

NGINX is an open source, high performance HTTP and reverse proxy server with many other web service-related features bundled. It is also used as a load balancer in the cloud. NGINX implements an asynchronous event-driven architecture to handle incoming requests and is built to offer a low memory footprint and high concurrency. NGINX is the most popular web server among high-traffic websites as of 20211.

The purpose of this guide is to describe techniques to run NGINX in an optimal manner on Ampere^® Altra^® processors.

The NGINX workload is comprised of a target running the NGINX web server and client(s) running the Wrk HTTP traffic generation program.

Running an application in a performant manner starts with building it correctly and using the appropriate compiler flags. When running on Ampere^® Altra^® processors, we recommend building from source with the GCC compiler version 10 or newer. Newer compilers tend to have better support for new processor features and incorporate more advanced code generation techniques.

We have used CentOS8 as the operating system for our testing.

• Download and install GCC 10 from SCL repository:

Copy
sudo yum -y install yum install scl-utils scl-utils-build 
sudo yum -y install gcc-toolset-10-gcc 
scl enable gcc-toolset-10 bash 

For other operating systems like Ubuntu 20.04 LTS and Debian, GCC 10.2.1 is available and can be installed directly from the respective repositories.

NGINX can be installed from repositories that the OS package manager offers or can be built directly from source. A comprehensive NGINX installation guide can be found in the official documentation. We recommend installing from source for flexibility, control, and the ability to configure specific modules.

To build NGINX optimized for Ampere^® Altra^® processors family, additional compile flags which can leverage hardware features can be added during compilation stage. NGINX source code for compilation can be obtained from NGINX download page. The stable version NGINX1.20.0 is used in our example. Installation from the source requires certain libraries and additional modules that will be compiled into the binary.

• Execute the following steps to install the dependencies.

Copy
yum install -y zlib-devel pcre-devel git wget
yum -y groupinstall "Development Tools"
# needed for openssl 3.x compilation
yum install perl-IPC-Cmd

NGINX, at its core, is a collection of modules. One can pick and choose modules depending on the functionality needed.

• To support https connections, download the latest code for these additional sources from the respective git repositories.

Copy
git clone https://github.com/openssl/openssl.git
git clone https://github.com/simpl/ngx_devel_kit.git

• Download NGINX source.

Copy
NGINX_VER=1.20.0 
wget http://nginx.org/download/nginx-$NGINX_VER.tar.gz  
tar -xvf nginx-$NGINX_VER.tar.gz 

The configure options are specified with the ./configure script in the source root directory. It allows for configuration of various NGINX parameters including the paths to the sources, compiler options, and the list of modules. The script also creates a makefile needed for the next step - compilation.

• Nonstandard compiler paths and additional compiler options can be set using the CC environmental variable.

Copy
# Note: set NGINX_DIR variable for the NGINX installation destination 
./configure \
     --with-http_ssl_module \
     --with-http_stub_status_module \
     --with-openssl=../openssl \
     --with-http_v2_module \
     --add-module=../ngx_devel_kit \
     --prefix=$NGINX_DIR
     

make -j`nproc` install  

In our example, NGINX is configured to support https and gzip compression. Please refer to the nginx.conf and gzip.conf files shown in the appendix to configure the server.

• Create your own ssl keys using OpenSSL and copy it to the system under test.

Copy
sudo cp key/NGINX_TEST_SSL.crt /etc/pki/tls/certs/
sudo cp key/NGINX_TEST_SSL.key /etc/pki/tls/private/
sudo cp key/NGINX_TEST_SSL.csr /etc/pki/tls/private/

• Start the server once NGINX is configured.

Copy
cp test.html  $NGINX_DIR/html
cp gzip.conf  $NGINX_DIR/conf/
cp nginx.conf $NGINX_DIR/conf/nginx.conf

#start the server
sudo $NGINX_DIR/sbin/nginx 

There are hundreds of settings that can alter the functionality and performance of NGINX. What is listed below are just some of the more common knobs that can be used. The NGINX documentation is the recommended resource to understand all the settings.

nginx conf file

Copy

# nginx.conf 
#   For more information on configuration, see:
#   * Official English Documentation: http://nginx.org/en/docs/
#   * Official Russian Documentation: http://nginx.org/ru/docs/

user nginx;
worker_processes auto;
worker_rlimit_nofile 104857600;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

# Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;

events {
    use epoll;
    accept_mutex off;
    worker_connections 10240;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    #access_log  /var/log/nginx/access.log  main;


    access_log     off;
    open_file_cache max=10240000 inactive=60s;
    open_file_cache_valid 80s;
    open_file_cache_min_uses 1;
    keepalive_requests 100000000000;
    keepalive_timeout  300s;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    types_hash_max_size 4096;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    # Load modular configuration files from the /etc/nginx/conf.d directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/nginx/conf.d/*.conf;

    server {
        listen       80;
        listen       [::]:80;
        server_name  _;
        root         /usr/share/nginx/html;

        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;

        error_page 404 /404.html;
            location = /40x.html {
        }

        error_page 500 502 503 504 /50x.html;
            location = /50x.html {
        }
    }

    server {
        listen       443 ssl http2;
        listen       [::]:443 ssl http2;
        server_name  _;
        root         /usr/share/nginx/html;

        ssl_certificate "/etc/pki/tls/certs/NGINX_TEST_SSL.crt";
        ssl_certificate_key "/etc/pki/tls/private/NGINX_TEST_SSL.key";

        ssl_protocols              TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
        #ssl_ciphers ALL:!aNULL:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP;
        #ssl_ciphers "AES128+SHA256 NULL+SHA256 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS !MEDIUM !RC4";
        ssl_ciphers "AES128+SHA256 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS !MEDIUM !RC4";
	ssl_prefer_server_ciphers on;

        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;

        error_page 404 /404.html;
            location = /40x.html {
        }

        error_page 500 502 503 504 /50x.html;
            location = /50x.html {
        }
    }

gzip conf file

Copy
gzip on;
gzip_min_length  100;
gzip_buffers  8 32k;
gzip_types  text/plain text/css application/x-javascript text/xml application/xml text/javascript;
gzip_vary on;

Usage statistics of web servers: https://w3techs.com/technologies/overview/web_server