Setup Environment to Deploy AIStor on 8-Node Bare Metal Cluster

System Setting and Checking

Each node of the cluster, add grub setting ‘iommu.passthrough=1’ in /etc/default/grub and update its grub.

Copy
GRUB_CMDLINE_LINUX_DEFAULT="iommu.passthrough=1"
$ sudo update-grub2
$ sudo update-grub2

Each node of the cluster, set the CPU scaling governor to ‘performance’ and verify to make sure that the nodes are running at ‘performance’ mode.

Copy
$ echo performance | sudo tee /sys/devices/system/cpu/*/cpufreq/scaling_governor
$ sudo cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor | uniq -c

Each node of the cluster, check link status for all drives to make sure the PCIe speed and width detected correctly. In this example, assume that we use Micron drives.

Copy
$ sudo lspci -vvv | grep -i Micron -A 50 | grep -i LnkSta | grep -i Speed  <br>
LnkSta:	Speed 16GT/s (ok), Width x4 (ok) <br>
LnkSta:	Speed 16GT/s (ok), Width x4 (ok) <br>
LnkSta:	Speed 16GT/s (ok), Width x4 (ok) <br>
LnkSta:	Speed 16GT/s (ok), Width x4 (ok) <br>
LnkSta:	Speed 16GT/s (ok), Width x4 (ok) <br>
LnkSta:	Speed 16GT/s (ok), Width x4 (ok) <br>
LnkSta:	Speed 16GT/s (ok), Width x4 (ok) <br>
LnkSta:	Speed 16GT/s (ok), Width x4 (ok) <br>

Check to see if the Max Read Request set to 512 bytes for the drives

Copy
$ sudo lspci -vvv | grep -i Micron -A 50 | grep -i MaxReadReq
MaxPayload 512 bytes, MaxReadReq 512 bytes
MaxPayload 256 bytes, MaxReadReq 512 bytes
MaxPayload 512 bytes, MaxReadReq 512 bytes
MaxPayload 256 bytes, MaxReadReq 512 bytes
MaxPayload 512 bytes, MaxReadReq 512 bytes
MaxPayload 256 bytes, MaxReadReq 512 bytes
MaxPayload 512 bytes, MaxReadReq 512 bytes
MaxPayload 256 bytes, MaxReadReq 512 bytes

Prior to use the drives and run benchmark, you need to format all the drives on each of the 8x nodes.

Copy
$ sudo apt install nvme-cli

Create an executable shell script format_drives.sh. Depend on which NVMe using for boot drive, exclude it from the format_drives.sh Note in the example, exclude nvme4 because it is the boot drive.

Copy
$ cat format_drives.sh 
nvme format -s1 /dev/nvme0n1 &
nvme format -s1 /dev/nvme1n1 &
nvme format -s1 /dev/nvme2n1 &
nvme format -s1 /dev/nvme3n1 &
nvme format -s1 /dev/nvme5n1 &
nvme format -s1 /dev/nvme6n1 &
nvme format -s1 /dev/nvme7n1 &
nvme format -s1 /dev/nvme8n1

Networking and Firewalls

Each node should have full bidirectional network access to every other node in the deployment. The following command explicitly opens the default AIStor server API port 9000 and 9001 for each node running firewall

Copy
$ sudo firewall-cmd --permanent --zone=public --add-port=9000/tcp
$ sudo firewall-cmd --permanent --zone=public --add-port=9001/tcp
$ sudo firewall-cmd --reload

Sequential Hostnames

AIStor requires using expansion notation {x...y} to denote a sequential series of AIStor hosts when creating a server pool. AIStor therefore requires using sequentially numbered hostnames to represent each AIStor server process in the deployment.

Note: to change the hostname, use the command ‘hostnamectl set-hostname <new hostname>’

Local JBOD Storage with Sequential Mounts

MinIO strongly recommends direct-attached JBOD arrays with XFS-formatted disks for best performance. Ensure all nodes in the deployment use the same type (NVMe, SSD, or HDD) drive with identical capacity.

AIStor does not distinguish drive types and does not benefit from mixed storage types. Additionally, AIStor limits the size used per drive to the smallest drive in the deployment.

AIStor requires using expansion notation {x...y} to denote a sequential series of drives when creating the new deployment, where all nodes in the deployment have an identical set of mounted drives. AIStor also requires that the ordering of physical drives remain constant across restarts, such that a given mount point always points to the same formatted drive. MinIO therefore strongly recommends using /etc/fstab or a similar file-based mount configuration to ensure that drive ordering cannot change after a reboot.

For example, on each AIStor node, xfs-formatted drives as below, assume that there are 8 drives per node:

Copy
ampere@storage-node1:~# sudo mkfs.xfs /dev/nvme0n1 -L minio-data1
meta-data=/dev/nvme0n1           isize=512    agcount=4, agsize=117210902 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1    bigtime=0 inobtcount=0
data     =                       bsize=4096   blocks=468843606, imaxpct=5
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=228927, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
Discarding blocks...Done.

.......

ampere@storage-node1:~# sudo mkfs.xfs /dev/nvme8n1 -L minio-data8
meta-data=/dev/nvme8n1           isize=512    agcount=4, agsize=117210902 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1    bigtime=0 inobtcount=0
data     =                       bsize=4096   blocks=468843606, imaxpct=5
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=228927, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
Discarding blocks...Done.

Using /etc/fstab to ensure that drive ordering cannot change after a reboot. Edit /etc/fstab in each AIStor node:

Copy
$ sudo vi /etc/fstab
# <file system>  <mount point>        <type>  <options>         <dump>  <pass>
LABEL=minio-data1 /mnt/minio-data1  xfs     defaults,noatime        0 0
LABEL=minio-data2 /mnt/minio-data2  xfs     defaults,noatime        0 0
LABEL=minio-data3 /mnt/minio-data3  xfs     defaults,noatime        0 0
LABEL=minio-data4 /mnt/minio-data4  xfs     defaults,noatime        0 0
LABEL=minio-data5 /mnt/minio-data5  xfs     defaults,noatime        0 0
LABEL=minio-data6 /mnt/minio-data6  xfs     defaults,noatime        0 0
LABEL=minio-data7 /mnt/minio-data7  xfs     defaults,noatime        0 0
LABEL=minio-data8 /mnt/minio-data8  xfs     defaults,noatime        0 0

You can then specify the entire range of drives using the expansion notation /mnt/minio-data{1...8}. AIStor does not support arbitrary migration of a drive with existing data to a new mount position, whether intentional or as the result of OS-level behavior. Reboot all 8x minio nodes after editing /etc/fstab files. Check the /mnt folder:

Copy
ampere@storage-node1:~$ ls /mnt
minio-data1  minio-data2  minio-data3  minio-data4  minio-data5  minio-data6 minio-data7 minio-data8

.........


ampere@storage-node8:~$ ls /mnt
minio-data1  minio-data2  minio-data3  minio-data4  minio-data5  minio-data6 minio-data7 minio-data8

Time Synchronization

Multi-node systems must maintain synchronized time and date to maintain stable internode operations and interactions. Make sure all nodes sync to the same time server regularly using ntp synchronization service.

Check on each AIStor node:

Copy
ampere@storage-node1:~$ timedatectl list-timezones | grep America/Los_Angeles
America/Los_Angeles
ampere@storage-node1:~$ sudo timedatectl set-timezone America/Los_Angeles
[sudo] password for ubuntu: 
ampere@storage-node1:~$ timedatectl
               Local time: Wed 2023-08-23 17:15:25 PDT
           Universal time: Thu 2023-08-24 00:15:25 UTC
           RTC time: Thu 2023-08-24 00:15:26

                Time zone: America/Los_Angeles (PDT, -0700)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

.......


ampere@storage-node8:~$ sudo timedatectl set-timezone America/Los_Angeles
[sudo] password for ubuntu: 
ampere@storage-node8:~$ timedatectl
               Local time: Wed 2023-08-23 17:17:07 PDT
           Universal time: Thu 2023-08-24 00:17:07 UTC
                 RTC time: Thu 2023-08-24 00:17:07
                Time zone: America/Los_Angeles (PDT, -0700)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

Homogeneous Node Configurations

MinIO strongly recommends selecting substantially similar hardware configurations for all nodes in the deployment. Ensure the hardware (CPU, memory, motherboard, storage adapters) and software (operating system, kernel settings, system services) is consistent across all nodes.

Install the AIStor Binary on Each Node

Use the following option to download the AIStor server installation file on each node running Ubuntu 22.04:

Copy
$ wget https://dl.min.io/aistor/minio/release/linux-arm64/archive/minio_20250407200512.0.0_arm64.deb -O minio.deb
$ sudo dpkg -i minio.deb

For example, on each minio node:

Copy
ampere@storage-node1:~$ sudo dpkg -i minio.deb
 
.....

ampere@storage-node8:~$ sudo dpkg -i minio.deb

Create the systemd Service File

The .deb packages install the following systemd service file to /etc/systemd/system/minio.service. For binary installations, create this file manually on all AIStor hosts:

Copy
[Unit]
Description=MinIO
Documentation=https://docs.min.io
Wants=network-online.target
After=network-online.target
AssertFileIsExecutable=/usr/local/bin/minio

[Service]
Type=notify

WorkingDirectory=/usr/local

User=root
Group=root
ProtectProc=invisible

EnvironmentFile=-/etc/minio/minio.config
ExecStartPre=/bin/bash -c "if [ -z \"${MINIO_VOLUMES}\" ]; then echo \"Variable MINIO_VOLUMES not set in /etc/default/minio\"; exit 1; fi"
ExecStart=/usr/local/bin/minio server $MINIO_OPTS $MINIO_VOLUMES

# Let systemd restart this service always
Restart=always

# Specifies the maximum file descriptor number that can be opened by this process
LimitNOFILE=1048576

# Specifies the maximum number of threads this process can create
TasksMax=infinity

# Disable timeout logic and wait until process is stopped
TimeoutStopSec=infinity
SendSIGKILL=no

[Install]
WantedBy=multi-user.target

# Built for ${project.name}-${project.version} (${project.name})

The minio.service file runs as the minio-user User and Group by default. You can create the user and group using the groupadd and useradd commands. The following example creates the user, group, and sets permissions to access the folder paths intended for use by AIStor. These commands typically require root (sudo) permissions.

Copy
$ sudo groupadd -r root
$ sudo useradd -M -r -g root root
$ sudo chown root:root /mnt/minio-data1 /mnt/minio-data2 /mnt/minio-data3 /mnt/minio-data4 /mnt/minio-data5  /mnt/minio-data6 /mnt/minio-data7 /mnt/minio-data8

Checking /mnt folder on ALL 8x AIStor nodes:

Copy
ampere@storage-node1:~$ ls -l /mnt/
total 0
drwxr-xr-x 5 root root 64 Apr 15 18:32 minio-data1
drwxr-xr-x 5 root root 64 Apr 15 18:32 minio-data2
drwxr-xr-x 5 root root 64 Apr 15 18:32 minio-data3
drwxr-xr-x 5 root root 64 Apr 15 18:32 minio-data4
drwxr-xr-x 5 root root 64 Apr 15 18:32 minio-data5
drwxr-xr-x 5 root root 64 Apr 15 18:32 minio-data6
drwxr-xr-x 5 root root 64 Apr 15 18:32 minio-data7
drwxr-xr-x 5 root root 64 Apr 15 18:32 minio-data8

.....

ampere@storage-node8:~$ ls -l /mnt/
total 0
drwxr-xr-x 5 root root 64 Apr 15 18:32 minio-data1
drwxr-xr-x 5 root root 64 Apr 15 18:32 minio-data2
drwxr-xr-x 5 root root 64 Apr 15 18:32 minio-data3
drwxr-xr-x 5 root root 64 Apr 15 18:32 minio-data4
drwxr-xr-x 5 root root 64 Apr 15 18:32 minio-data5
drwxr-xr-x 5 root root 64 Apr 15 18:32 minio-data6
drwxr-xr-x 5 root root 64 Apr 15 18:32 minio-data7
drwxr-xr-x 5 root root 64 Apr 15 18:32 minio-data8

Create the Service Environment File

Create an environment file at /etc/default/minio. The AIStor service uses this file as the source of all environment variables used by AIStor and the minio.service file. The following example assumes that the deployment has a single server pool consisting of:

• Eight AIStor server hosts with sequential hostnames: storage-node1, storage-node2, storage-node3, storage-node4, storage-node5, storage-node6, storage-node7, storage-node8
• All hosts have eight locally-attached drives with sequential mount-points:

Copy
ampere@storage-node1:~$ ls -l /mnt/
total 0
drwxr-xr-x 5 root root 64 Apr 15 18:32 minio-data1
drwxr-xr-x 5 root root 64 Apr 15 18:32 minio-data2
drwxr-xr-x 5 root root 64 Apr 15 18:32 minio-data3
drwxr-xr-x 5 root root 64 Apr 15 18:32 minio-data4
drwxr-xr-x 5 root root 64 Apr 15 18:32 minio-data5
drwxr-xr-x 5 root root 64 Apr 15 18:32 minio-data6
drwxr-xr-x 5 root root 64 Apr 15 18:32 minio-data7
drwxr-xr-x 5 root root 64 Apr 15 18:32 minio-data8

Modify the example to reflect your deployment topology. In the below example, the environment file at /etc/default/minio is created for 8-node cluster with 8-drives.

Copy
# Set the hosts and volumes AIStor uses at startup
# The command uses AIStor expansion notation {x...y} to denote a
# sequential series.
#
# The following example covers four AIStor hosts
# with 8 drives each at the specified hostname and drive locations.
# The command includes the port that each AIStor server listens on
# (default 9000)

MINIO_VOLUMES="http://storage-node{1...8}:9000/mnt/minio-data{1...8}"

# Set all AIStor server options
#
# The following explicitly sets the AIStor Console listen address to
# port 9001 on all network interfaces. The default behavior is dynamic
# port selection.

MINIO_OPTS="--console-address :9001"

# Set the root username. This user has unrestricted permissions to
# perform S3 and administrative API operations on any resource in the
# deployment.
#
# Defer to your organizations requirements for superadmin user name.

MINIO_ROOT_USER=<minio-user>

# Set the root password
#
# Use a long, random, unique string that meets your organizations
# requirements for passwords.

MINIO_ROOT_PASSWORD=<minio-password>

# Set to the URL of the load balancer for the AIStor deployment
# This value *must* match across all AIStor servers. If you do
# not have a load balancer, set this value to any *one* of the
# AIStor hosts in the deployment as a temporary measure.
MINIO_SERVER_URL="http://192.168.4.201:9000"

NOTE: Put your own MINIO_ROOT_USER and MINIO_ROOT_PASSWORD

MINIO_VOLUME settings for different AIStor topology as an example

Run the AIStor Server Process

Issue the following commands on each node in the deployment to start the AIStor service:

Copy
$ sudo systemctl start minio.service

Use the following commands to confirm the service is online and functional on each node:

Copy
$ sudo systemctl status minio.service
$ sudo systemctl enable minio
Created symlink /etc/systemd/system/multi-user.target.wants/minio.service → /etc/systemd/system/minio.service.

ampere@storage-node1:~$ sudo journalctl -f -u minio.service
Aug 24 11:17:00 minio1 minio[82802]: Automatically configured API requests per node based on available memory on the system: 2711
Aug 24 11:17:00 minio1 minio[82802]: All MinIO sub-systems initialized successfully in 4.408856885s
Aug 24 11:17:00 minio1 minio[82802]: MinIO Object Storage Server
Aug 24 11:17:00 minio1 minio[82802]: Copyright: 2015-2025 MinIO, Inc.


......


ampere@storage-node8:~$ sudo journalctl -f -u minio.service
Aug 24 11:16:56 minio2 minio[78231]: All MinIO sub-systems initialized successfully in 26.722115ms
Aug 24 11:16:56 minio2 minio[78231]: MinIO Object Storage Server
Aug 24 11:16:56 minio2 minio[78231]: Copyright: 2015-2025 MinIO, Inc.

Note: If any drives remain offline after starting AIStor, check and cure any issues blocking their functionality before starting production workloads.

Shut down the AIStor Hosts

Copy
$ sudo systemctl stop minio.service

Note: if you need to create another AIStor topology using different number of servers and drives, you need to delete the AIStor resources prior to start and enable AIStor setup again.

Copy
$ sudo rm -rf /mnt/disk*/minio/.minio.sys
$ ls -la  /mnt/disk*/minio

MinIO helps you measure the throughput of your servers under various conditions by simulating read and write workloads on specified numbers of objects with different sizes. Warp is a performance benchmarking tool specifically designed for distributed S3 service including evaluating their scalability and latency under heavy workload. It is particularly useful when testing large-scale object storage deployments. Warp tests consist of several components:

• Coordinator node which is responsible for managing the overall benchmarking processes and distributing workloads among worker nodes and collecting performance data from them.
• Worker node(s) which execute various read/write requests against AIStor servers in parallel based on instructions received from the master node.
• Clients which generate I/O traffic by sending random read/write requests using a custom client library provided with Warp test. These clients interact only with worker nodes, not directly with servers, to help isolate any impact of network latency on performance measurements.

In this reference architecture, we run a series of GET, LIST, PUT, DELETE and STAT operations tested by MinIO Warp. These tests are crucial for AI inference applications because these operations directly impact the speed, reliability, and scalability of the data pipeline.

PUT tests: AI inference pipelines often require uploading large batches of videos, images, audios, or sensor data before they can be processed. High PUT performance ensures fast ingestion from data preprocessing pipelines into object storage.

GET tests: AI models (e.g.YOLO, Whisper) need to pull data for real time or batch inference. Fast and scalable GET performance means quick access to input data, reducing inference latency and maximizing CPU utilization.

LIST tests: Listing objects are used to dynamically determine which data are available for inference or post-processing. Efficient LIST operations enable real-time workload scheduling and queueing for AI tests.

DELETE tests: to avoid storage bloat, AI pipelines frequently delete low-priority data. Fast DELETE operations reduce storage cost and keep the pipeline clean, and efficient, especially in long-running inference workflows.

STAT tests: STAT is used to check existence, size, or last modified time of files (Metadata lookup) - useful in version control, model checkpoints, etc. High performance STAT operations help orchestrators make fast decisions about data readiness.

Client Warp Benchmark

Warp test is highly configurable and can customize the number of clients, concurrency levels, object sizes, test durations, and more to accurately assess AIStor’s performance under different conditions.

Download warp test on each of the 8x AIStor clients

Copy
$ wget ‘https://github.com/minio/warp/releases/download/v1.3.0/warp_Linux_arm64.deb’

Install warp deb package on each of the 8x AIStor clients

Copy
$ sudo dpkg -i warp_Linux_arm64.deb 

Run warp client on each of the 8x AIStor clients

Copy
$ warp client
warp: Listening on :7761

Run warp benchmark example.

Copy
$ warp get --duration=1m --warp-client=minio-client{1...8} --host=storage-node{1...8}:9000 --access-key=<access-key> --secret-key=<secret-key> --bucket=smc-8nodes-cluster-warptest --obj.size=64M

Network Bandwidth Consideration

When testing AIStor performance using Warp, it is important to consider the impact of network bandwidth on results because object storage systems often involve transferring large amounts of data over a network.

Here are some of the factors, considerations and best practices related to networking when conducting AIStor benchmarking:

• Consider using dedicated test networks with sufficient capacity for minimizing the impact of contention on performance measurements when testing on large scale, distributing environment, or large number of drives per nodes, or HDD vs. SSD drives.
• Consider using tools like iPerf to measure and analyze your test network’s bandwidth capabilities independently of AIStor benchmarks for better understanding the impact on overall system performance

GET Test

Run a 5-minute benchmark test simulating 100 concurrent GET requests per client across 8 clients (800 concurrent in total), randomly retrieving 125,000 objects of 10 KiB, 8MiB, and 64MiB each from the warp-bench bucket over encrypted TLS.

PUT Test

Run a 5-minute benchmark test simulating 100 concurrent PUT requests per client across 8 clients (800 concurrent in total), objects of 10 KiB, 8MiB, and 64MiB each from the warp-bench bucket over encrypted TLS.

DELETE Test

Run a 5-minute benchmark test that simulates deleting 125,000 objects of approximately 10 KiB,8MiB, and 64MiB each, using 100 concurrent delete requests per client across 8 clients (800 concurrent in total). It targets objects with the specified prefix in the warp-bench bucket over a encrypted TLS connection

LIST Test

Run a 5-minute benchmark that simulates listing objects (with a specific prefix) in the warp-bench bucket using 100 concurrent LIST requests per client across 8 clients (800 concurrent in total). It tests the LIST operation performance over a secure/encrypted TLS connection

STAT Test

Run a 5-minute benchmark that simulates retrieving metadata (STAT) for objects with a specific prefix in the warp-bench bucket. It performs 100 concurrent STAT requests per client across 8 clients (800 concurrent in total) over a secure encrypted TLS connection.

GET Test

Run a 5-minute benchmark test simulating 100 concurrent GET requests per client across 8 clients (800 concurrent in total), randomly retrieving 125,000 objects of 10 KiB, 8MiB, and 64MiB each from the warp-bench bucket over unencrypted connection.

PUT Test

Run a 5-minute benchmark test simulating 100 concurrent PUT requests per client across 8 clients (800 concurrent in total), objects of 10 KiB, 8MiB, and 64MiB each from the warp-bench bucket over unencrypted connection.

DELETE Test

Run a 5-minute benchmark test that simulates deleting 125,000 objects of approximately 10 KiB,8MiB, and 64MiB each, using 100 concurrent delete requests per client across 8 clients (800 concurrent in total). It targets objects with the specified prefix in the warp-bench bucket over an unencrypted connection

LIST Test

Run a 5-minute benchmark that simulates listing objects (with a specific prefix) in the warp-bench bucket using 100 concurrent LIST requests per client across 8 clients (800 concurrent in total). It tests the LIST operation performance over a unencrypted connection

STAT Test

Run a 5-minute benchmark that simulates retrieving metadata (STAT) for objects with a specific prefix in the warp-bench bucket. It performs 100 concurrent STAT requests per client across 8 clients (800 concurrent in total) over a secure unencrypted connection.

The results from each of the tests are shown below.

The reference architecture and the solution presented in this paper are focused on running AIStor on a multi-node cluster with Ampere processors. The enterprise license of AIStor was run on Ampere processors and no issues were discovered in the test cases. Customers can transition their existing AIStor clusters running x86 to Ampere-based systems without encountering major downtime disruptions. This solution offers a high-performance, scalable, and efficient storage backend ideal for modern AI, analytics, and cloud-native workloads. The results of our benchmark tests confirm that the collaboration between MinIO, Supermicro®, Micron, and Ampere® delivers a powerful, scalable, and high-performance solution for AI-driven workloads.

About MinIO and Ampere® Computing

MinIO is the data foundation for enterprise AI. Built for exascale performance and limitless scale, MinIO AIStor delivers a secure, sovereign, and AI-readydata store that spans from edge to core to cloud. With rampant adoption across the Fortune 100 and 500, MinIO is redefining how organizations and government agencies store, manage, and mobilize all their data in the AI era. MinIO is backed by Jerry Yang’s AME Cloud Venture, Dell Technologies, General Catalyst, Index Ventures, Intel Capital, Softbank Vision Fun 2, and others.

Ampere® Computing is a semiconductor design company for a new era, leading the future of computing with an innovative approach to CPU design focused on high-performance, energy efficient AI compute. As a pioneer in the new frontier of energy efficient high-performance computing, Ampere® is part of the Softbank Group of companies driving sustainable computing for AI, Cloud, and edge applications.

To learn more about our developer efforts and find best practices, visit Ampere’s Developer Center and join the conversation in the Ampere Developer Community.