LINBIT SDS on Ampere Altra Max microK8s Cluster
on Ampere® Altra® Max microK8s Cluster
Overview
This tutorial contains 3 sections. The first is a step-by-step tutorial on how to deploy a 3-node MicroK8s cluster on Ampere Altra Max platforms. The second section shows how to install and configure LINSTOR, and industry-leading software defined storage (SDS) package developed by LINBIT. Lastly, the document shows how to set up and execute system performance tests.
It’s worth mentioning that the concurrent random read I/O operations per second measurement was the highest LINBIT achieved on a benchmarking platform at the time of this writing. More importantly, it was achieved on a cluster with 75% fewer systems than the previous record. This density is the key to Ampere’s unmatched rack density and performance per rack.
Prerequisites
System Specifications
Each of the three Ampere systems were identical and equipped with the following hardware:
-
Processor: 1x Ampere® Altra® Max 128-core AArch64 processor
-
Memory: 250GiB DDR4-3200
-
Storage:
- 8x PCIe Gen.4 Samsung PM1733a SSDs (30.72TB)
- 1x PCIe Gen.3 Samsung PM983 SSD (960GB)
-
Network: 2x Mellanox MT28908 [ConnectX-6] 100Gbe
-
Operating System: Ubuntu 22.04
System Specifications
Network Configuration
| Hostname | Management and MicroK8s CNI IP Address | LINSTOR Replication IP Address |
|---|---|---|
| linbit1 | 192.168.4.195 | 10.10.10.101 |
| linbit2 | 192.168.4.198 | 10.10.10.102 |
| linbit3 | 192.168.4.56 | 10.10.10.103 |
3-Node MicroK8s Setup
MicroK8s Installation and Configuration
MicroK8s (distributed by Canonical), was installed and configured as described below. Install MicroK8s from snap on all three Ampere systems:
ampere@linbit1:~$ sudo snap install microk8s --classic Run configure hook of "microk8s" snap if present microk8s (1.26/stable) v1.26.0 from Canonical✓ installed ampere@linbit2:~$ sudo snap install microk8s --classic microk8s (1.26/stable) v1.26.0 from Canonical✓ installed ampere@linbit3:~$ sudo snap install microk8s --classic microk8s (1.26/stable) v1.26.0 from Canonical✓ installed
Add your user to the MicroK8s group so that the user can run MicroK8s commands without needing elevated privileges:
ampere@linbit1:~$ sudo usermod -a -G microk8s $USER ampere@linbit2:~$ sudo usermod -a -G microk8s $USER ampere@linbit3:~$ sudo usermod -a -G microk8s $USER
Change the ownership recursively on the MicroK8s directory. By default, this is the .kube directory created in the installing user’s home directory:
ampere@linbit1:~$ sudo chown -f -R $USER ~/.kube ampere@linbit2:~$ sudo chown -f -R $USER ~/.kube ampere@linbit3:~$ sudo chown -f -R $USER ~/.kube
Log in again to have the user’s group membership change take effect:
ampere@linbit1:~$ su - $USER ampere@linbit2:~$ su - $USER ampere@linbit3:~$ su - $USER
Verifying MicroK8s Installation
Entering a status command should show that MicroK8s is running and show the enabled and disabled add-ons on each node.
Notice the message about MircoK8s’s high-availability status. You will eventually change its status to “yes” , when you join nodes together, later in this tutorial.
ampere@linbit1:~$ microk8s status --wait-ready microk8s is running high-availability: no datastore master nodes: 127.0.0.1:19001 datastore standby nodes: none addons: enabled: ha-cluster # (core) Configure high availability on the current node helm # (core) Helm - the package manager for Kubernetes helm3 # (core) Helm 3 - the package manager for Kubernetes disabled: cert-manager # (core) Cloud native certificate management community # (core) The community addons repository dashboard # (core) The Kubernetes dashboard dns # (core) CoreDNS host-access # (core) Allow Pods connecting to Host services smoothly hostpath-storage # (core) Storage class; allocates storage from host directory ingress # (core) Ingress controller for external access kube-ovn # (core) An advanced network fabric for Kubernetes mayastor # (core) OpenEBS MayaStor metallb # (core) Loadbalancer for your Kubernetes cluster metrics-server # (core) K8s Metrics Server for API access to service metrics minio # (core) MinIO object storage observability # (core) A lightweight observability stack for logs, traces and metrics prometheus # (core) Prometheus operator for monitoring and logging rbac # (core) Role-Based Access Control for authorisation registry # (core) Private image registry exposed on localhost:32000 storage # (core) Alias to hostpath-storage add-on, deprecated ampere@linbit2:~$ microk8s status --wait-ready microk8s is running high-availability: no datastore master nodes: 127.0.0.1:19001 datastore standby nodes: none addons: enabled: ha-cluster # (core) Configure high availability on the current node helm # (core) Helm - the package manager for Kubernetes helm3 # (core) Helm 3 - the package manager for Kubernetes disabled: cert-manager # (core) Cloud native certificate management community # (core) The community addons repository dashboard # (core) The Kubernetes dashboard dns # (core) CoreDNS host-access # (core) Allow Pods connecting to Host services smoothly hostpath-storage # (core) Storage class; allocates storage from host directory ingress # (core) Ingress controller for external access kube-ovn # (core) An advanced network fabric for Kubernetes mayastor # (core) OpenEBS MayaStor metallb # (core) Loadbalancer for your Kubernetes cluster metrics-server # (core) K8s Metrics Server for API access to service metrics minio # (core) MinIO object storage observability # (core) A lightweight observability stack for logs, traces and metrics prometheus # (core) Prometheus operator for monitoring and logging rbac # (core) Role-Based Access Control for authorisation registry # (core) Private image registry exposed on localhost:32000 storage # (core) Alias to hostpath-storage add-on, deprecated ampere@linbit3:~$ microk8s status --wait-ready microk8s is running high-availability: no datastore master nodes: 127.0.0.1:19001 datastore standby nodes: none addons: enabled: ha-cluster # (core) Configure high availability on the current node helm # (core) Helm - the package manager for Kubernetes helm3 # (core) Helm 3 - the package manager for Kubernetes disabled: cert-manager # (core) Cloud native certificate management community # (core) The community addons repository dashboard # (core) The Kubernetes dashboard dns # (core) CoreDNS host-access # (core) Allow Pods connecting to Host services smoothly hostpath-storage # (core) Storage class; allocates storage from host directory ingress # (core) Ingress controller for external access kube-ovn # (core) An advanced network fabric for Kubernetes mayastor # (core) OpenEBS MayaStor metallb # (core) Loadbalancer for your Kubernetes cluster metrics-server # (core) K8s Metrics Server for API access to service metrics minio # (core) MinIO object storage observability # (core) A lightweight observability stack for logs, traces and metrics prometheus # (core) Prometheus operator for monitoring and logging rbac # (core) Role-Based Access Control for authorisation registry # (core) Private image registry exposed on localhost:32000 storage # (core) Alias to hostpath-storage add-on, deprecated
Verify the pods that are up and running by entering the following command:
ampere@linbit1:~$ microk8s.kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system calico-node-4hppc 1/1 Running 0 94m kube-system calico-kube-controllers-776f86ffd5-j4pm9 1/1 Running 0 94m ampere@linbit2:~$ microk8s.kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system calico-kube-controllers-69c494fc5-mz4xv 1/1 Running 0 95m kube-system calico-node-v4lr6 1/1 Running 0 95m ampere@linbit3:~$ microk8s.kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system calico-node-42wjd 1/1 Running 0 95m kube-system calico-kube-controllers-844577968-46c8x 1/1 Running 0 95m
Enabling High Availability on MicroK8s
MicroK8s offers is the ability to join nodes together. When at least three nodes have joined together, MicroK8s will automatically make itself highly available.
Before you join your nodes together, you need to verify that your nodes’ host names resolve to their respective IP addresses in the network on which you will be joining your nodes.
LINSTOR Installation and Configuration in 3-Node MicroK8s
The nodes have the following lines in each of their /etc/hosts file:
192\. 168.4.195 linbit1
192\. 168.4.198 linbit2
192\. 168.4.56 linbit3 Prior to join nodes together, check each node and make sure that its status is ‘ready’:
ampere@linbit1:~$ microk8s.kubectl get nodes
NAME STATUS ROLES AGE VERSION
linbit1 Ready <none> 17h v1.26.0
ampere@linbit2:~$ microk8s.kubectl get nodes
NAME STATUS ROLES AGE VERSION
linbit2 Ready <none> 17h v1.26.0
ampere@linbit3:~$ microk8s.kubectl get nodes
NAME STATUS ROLES AGE VERSION
linbit3 Ready <none> 17h v1.26.0 Enter the following command on one of your nodes, to generate a token for joining nodes. On the 1st node (linbit1), run the command microk8s add-node to generate a token:
ampere@linbit1:~$ microk8s add-node
From the node you wish to join to this cluster, run the following:
microk8s join 192.168.4.195:25000/bf8a790e4d9cb7795a27d2289b97cd4b/b75751ce8f4d
Use the '--worker' flag to join a node as a worker not running the control plane, eg:
microk8s join 192.168.4.195:25000/bf8a790e4d9cb7795a27d2289b97cd4b/b75751ce8f4d --worker
If the node you are adding is not reachable through the default interface you can use one of the following:
microk8s join 192.168.4.195:25000/bf8a790e4d9cb7795a27d2289b97cd4b/b75751ce8f4d
On the 2nd node (linbit2), run this command to join it to the cluster:
ampere@linbit2:~$ microk8s join 192.168.4.195:25000/bf8a790e4d9cb7795a27d2289b97cd4b/b75751ce8f4d WARNING: Hostpath storage is enabled and is not suitable for multi node clusters.cd4b/b75751ce8f4d Contacting cluster at 192.168.4.195 Waiting for this node to finish joining the cluster. .. .. .. .
Once completed, check the cluster’s status:
ampere@linbit1:~$ microk8s.kubectl get nodes NAME STATUS ROLES AGE VERSION linbit2 Ready <none> 5m18s v1.26.0 linbit1 Ready <none> 17h v1.26.0
Repeat above steps to generate a token for joining node 3 (linbit3):
ampere@linbit1:~$ microk8s add-node
From the node you wish to join to this cluster, run the following:
microk8s join 192.168.4.195:25000/794554fd04d714a9eca7ba6ba0cdda0f/b75751ce8f4d
Use the '--worker' flag to join a node as a worker not running the control plane, eg:
microk8s join 192.168.4.195:25000/794554fd04d714a9eca7ba6ba0cdda0f/b75751ce8f4d --worker
If the node you are adding is not reachable through the default interface you can use one of the following:
microk8s join 192.168.4.195:25000/794554fd04d714a9eca7ba6ba0cdda0f/b75751ce8f4d ampere@linbit3:~$ microk8s join 192.168.4.195:25000/794554fd04d714a9eca7ba6ba0cdda0f/b75751ce8f4d WARNING: Hostpath storage is enabled and is not suitable for multi node clusters.da0f/b75751ce8f4d Contacting cluster at 192.168.4.195 Waiting for this node to finish joining the cluster. .. .. ..
Once completed, check the cluster’s status. You should see 3 nodes now:
ampere@linbit1:~$ microk8s.kubectl get nodes NAME STATUS ROLES AGE VERSION linbit1 Ready <none> 17h v1.26.0 linbit2 Ready <none> 7m40s v1.26.0 linbit3 Ready <none> 16m v1.26.0
You can use a status command from any node to verify the proper joining of nodes and that MicroK8s has high availability enabled.
ampere@linbit1:~$ microk8s status microk8s is running high-availability: yes datastore master nodes: 192.168.4.195:19001 192.168.4.198:19001 192.168.4.56:19001 datastore standby nodes: none addons: enabled: ha-cluster # (core) Configure high availability on the current node helm # (core) Helm - the package manager for Kubernetes helm3 # (core) Helm 3 - the package manager for Kubernetes storage # (core) Alias to hostpath-storage add-on, deprecated disabled: cert-manager # (core) Cloud native certificate management community # (core) The community addons repository dashboard # (core) The Kubernetes dashboard host-access # (core) Allow Pods connecting to Host services smoothly ingress # (core) Ingress controller for external access kube-ovn # (core) An advanced network fabric for Kubernetes mayastor # (core) OpenEBS MayaStor metallb # (core) Loadbalancer for your Kubernetes cluster metrics-server # (core) K8s Metrics Server for API access to service metrics minio # (core) MinIO object storage observability # (core) A lightweight observability stack for logs, traces and metrics prometheus # (core) Prometheus operator for monitoring and logging rbac # (core) Role-Based Access Control for authorisation registry # (core) Private image registry exposed on localhost:32000
LINSTOR Installation and Configuration
LINSTOR Deployment and Configuration
To use the LINSTOR Operator’s compile mode for DRBD kernel module loading, you must install the ‘build- essential’ package on each of the cluster nodes:
ubuntu@linbit1:~$ sudo apt install build-essential [sudo] password for ubuntu: Reading package lists... Done Building dependency tree... Done Reading state information... Done ..... Scanning linux images... Running kernel seems to be up-to-date. Failed to check for processor microcode upgrades. No services need to be restarted. No containers need to be restarted. No user sessions are running outdated binaries. No VM guests are running outdated hypervisor (qemu) binaries on this host. ubuntu@linbit-2:~$ sudo apt install build-essential [sudo] password for ubuntu: Reading package lists... Done Building dependency tree... Done Reading state information... Done ..... Scanning linux images... Running kernel seems to be up-to-date. Failed to check for processor microcode upgrades. No services need to be restarted. No containers need to be restarted. No user sessions are running outdated binaries. No VM guests are running outdated hypervisor (qemu) binaries on this host. ubuntu@linbit-3:~$ sudo apt install build-essential [sudo] password for ubuntu: Reading package lists... Done Building dependency tree... Done Reading state information... Done ..... Scanning linux images... Running kernel seems to be up-to-date. Failed to check for processor microcode upgrades. No services need to be restarted. No containers need to be restarted. No user sessions are running outdated binaries. No VM guests are running outdated hypervisor (qemu) binaries on this host.
LINSTOR Operator v1 is installed using Helm. Enable the Helm and DNS services in MicroK8s from any node in the Kubernetes cluster:
ampere@linbit1:~$ microk8s.enable helm ampere@linbit2:~$ microk8s.enable helm ampere@linbit3:~$ microk8s.enable helm ampere@linbit1:~$ microk8s.enable dns ampere@linbit2:~$ microk8s.enable dns ampere@linbit3:~$ microk8s.enable dns
Create a Kubernetes secret named ‘drbdiocred’ using your http://my.linbit.com credentials:
ampere@linbit1:~$ microk8s.kubectl create secret docker-registry drbdiocred \ --docker-server=drbd.io --docker-username=$USER \ --docker-email=$EMAIL_ADDR --docker-password=$PASSWORD
Add the LINSTOR Helm repository from any node in the cluster:
ampere@linbit1:~$ microk8s.helm repo add linstor https://charts.linstor.io ampere@linbit1:~$ KUBECONFIG=/var/snap/microk8s/current/credentials/client.config
Create the LINSTOR Operator configuration file on any node in the cluster:
operator:
controller:
dbConnectionURL: k8s
satelliteSet:
storagePools:
lvmPools:
- name: lvm-thick-nvme0n1
volumeGroup: "thickpool-n0"
devicePaths:
- /dev/nvme0n1
- name: lvm-thick-nvme1n1
volumeGroup: "thickpool-n1"
devicePaths:
- /dev/nvme1n1
- name: lvm-thick-nvme3n1
volumeGroup: "thickpool-n3"
devicePaths:
- /dev/nvme3n1
- name: lvm-thick-nvme4n1
volumeGroup: "thickpool-n4"
devicePaths:
- /dev/nvme4n1
- name: lvm-thick-nvme5n1
volumeGroup: "thickpool-n5"
devicePaths:
- /dev/nvme5n1
- name: metapool-nvme6n1
volumeGroup: "metapool-n6"
devicePaths:
- /dev/nvme6n1
- name: lvm-thick-nvme7n1
volumeGroup: "thickpool-n7"
devicePaths:
- /dev/nvme7n1
- name: lvm-thick-nvme8n1
volumeGroup: "thickpool-n8"
devicePaths:
- /dev/nvme8n1
kernelModuleInjectionImage: drbd.io/arm64/drbd9-jammy
kernelModuleInjectionMode: Compile
etcd:
enabled: false
stork:
enabled: false
csi:
enableTopology: true
kubeletPath: "/var/snap/microk8s/common/var/lib/kubelet" The csi.kubeletPath value is required for MicroK8s when it is installed using snap.
Install LINSTOR Operator using Helm from the node where the LINSTOR Operator configuration file was created:
ampere@linbit1:~$ microk8s.helm install –kubeconfig $KUBECONFIG -f linstor-op-vals.yaml linstor-op linstor/linstor
For LINSTOR Operator v1, we must specify the replication network address for each node from within the LINSTOR controller pod:
ampere@linbit1:~$ microk8s.kubectl exec -it deployments/linstor-op-cs-controller – bash # linstor node interface create linbit1 rep_nic 10.10.10.101 # linstor node interface create linbit2 rep_nic 10.10.10.102 # linstor node interface create linbit3 rep_nic 10.10.10.103
Verifying LINSTOR Deployment
Wait for all resources to be ready.
ampere@linbit1:~$ microk8s.kubectl wait --namespace default --for=condition=Ready --timeout=10m pod --all pod/linstor-op-operator-589c968767-4t7jn condition met pod/linstor-op-cs-controller-75899d6697-jjmwd condition met pod/linstor-op-ns-node-f6dv7 condition met pod/linstor-op-csi-node-n4ftj condition met pod/linstor-op-ns-node-zwqnn condition met pod/linstor-op-csi-controller-697dc98569-lbqvj condition met pod/linstor-op-csi-node-xbn58 condition met pod/linstor-op-ns-node-729gk condition met pod/linstor-op-csi-node-l2w9w condition met
Check all the pods to make sure that they all ready and running.
ampere@linbit1:~$ microk8s.kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system calico-kube-controllers-776f86ffd5-j4pm9 1/1 Running 0 7d2h kube-system calico-node-pkcpd 1/1 Running 0 6d9h kube-system calico-node-dqm76 1/1 Running 0 6d9h kube-system calico-node-vkxxq 1/1 Running 1 (26h ago) 6d9h default linstor-op-operator-589c968767-4t7jn 1/1 Running 0 3m4s default linstor-op-cs-controller-75899d6697-jjmwd 1/1 Running 0 2m52s default linstor-op-ns-node-f6dv7 2/2 Running 0 2m52s default linstor-op-csi-node-n4ftj 3/3 Running 0 2m53s default linstor-op-ns-node-zwqnn 2/2 Running 0 2m52s default linstor-op-csi-controller-697dc98569-lbqvj 6/6 Running 0 2m52s default linstor-op-csi-node-xbn58 3/3 Running 0 2m53s default linstor-op-ns-node-729gk 2/2 Running 0 2m52s default linstor-op-csi-node-l2w9w 3/3 Running 0 2m53s ampere@linbit2:~$ microk8s.kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system calico-kube-controllers-776f86ffd5-j4pm9 1/1 Running 0 7d2h kube-system calico-node-pkcpd 1/1 Running 0 6d9h kube-system calico-node-dqm76 1/1 Running 0 6d9h kube-system calico-node-vkxxq 1/1 Running 1 (26h ago) 6d9h default linstor-op-operator-589c968767-4t7jn 1/1 Running 0 3m39s default linstor-op-cs-controller-75899d6697-jjmwd 1/1 Running 0 3m27s default linstor-op-ns-node-f6dv7 2/2 Running 0 3m27s default linstor-op-csi-node-n4ftj 3/3 Running 0 3m28s default linstor-op-ns-node-zwqnn 2/2 Running 0 3m27s default linstor-op-csi-controller-697dc98569-lbqvj 6/6 Running 0 3m27s default linstor-op-csi-node-xbn58 3/3 Running 0 3m28s default linstor-op-ns-node-729gk 2/2 Running 0 3m27s default linstor-op-csi-node-l2w9w 3/3 Running 0 3m28s ampere@linbit3:~$ microk8s.kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system calico-kube-controllers-776f86ffd5-j4pm9 1/1 Running 0 7d2h kube-system calico-node-pkcpd 1/1 Running 0 6d9h kube-system calico-node-dqm76 1/1 Running 0 6d9h kube-system calico-node-vkxxq 1/1 Running 1 (26h ago) 6d9h default linstor-op-operator-589c968767-4t7jn 1/1 Running 0 3m46s default linstor-op-cs-controller-75899d6697-jjmwd 1/1 Running 0 3m34s default linstor-op-ns-node-f6dv7 2/2 Running 0 3m34s default linstor-op-csi-node-n4ftj 3/3 Running 0 3m35s default linstor-op-ns-node-zwqnn 2/2 Running 0 3m34s default linstor-op-csi-controller-697dc98569-lbqvj 6/6 Running 0 3m34s default linstor-op-csi-node-xbn58 3/3 Running 0 3m35s default linstor-op-ns-node-729gk 2/2 Running 0 3m34s default linstor-op-csi-node-l2w9w 3/3 Running 0 3m35s
List the LINSTOR storage pools.
ampere@linbit1:~$ microk8s.kubectl exec -it deployments/linstor-op-cs-controller -- linstor storage-pool list ╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ ┊ StoragePool ┊ Node ┊ Driver ┊ PoolName ┊ FreeCapacity ┊ TotalCapacity ┊ CanSnapshots ┊ State ┊ SharedName ┊ ╞═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡ ┊ DfltDisklessStorPool ┊ linbit1 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊ ┊ ┊ DfltDisklessStorPool ┊ linbit2 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊ ┊ ┊ DfltDisklessStorPool ┊ linbit3 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊ ┊ ┊ lvm-thick-nvme0n1 ┊ linbit1 ┊ LVM ┊ thickpool-n0 ┊ 27.94 TiB ┊ 27.94 TiB ┊ False ┊ Ok ┊ ┊ ┊ lvm-thick-nvme0n1 ┊ linbit2 ┊ LVM ┊ thickpool-n0 ┊ 27.94 TiB ┊ 27.94 TiB ┊ False ┊ Ok ┊ ┊ ┊ lvm-thick-nvme0n1 ┊ linbit3 ┊ LVM ┊ thickpool-n0 ┊ 27.94 TiB ┊ 27.94 TiB ┊ False ┊ Ok ┊ ┊ ┊ lvm-thick-nvme1n1 ┊ linbit1 ┊ LVM ┊ thickpool-n1 ┊ 27.94 TiB ┊ 27.94 TiB ┊ False ┊ Ok ┊ ┊ ┊ lvm-thick-nvme1n1 ┊ linbit2 ┊ LVM ┊ thickpool-n1 ┊ 27.94 TiB ┊ 27.94 TiB ┊ False ┊ Ok ┊ ┊ ┊ lvm-thick-nvme1n1 ┊ linbit3 ┊ LVM ┊ thickpool-n1 ┊ 27.94 TiB ┊ 27.94 TiB ┊ False ┊ Ok ┊ ┊ ┊ lvm-thick-nvme3n1 ┊ linbit1 ┊ LVM ┊ thickpool-n3 ┊ 27.94 TiB ┊ 27.94 TiB ┊ False ┊ Ok ┊ ┊ ┊ lvm-thick-nvme3n1 ┊ linbit2 ┊ LVM ┊ thickpool-n3 ┊ 27.94 TiB ┊ 27.94 TiB ┊ False ┊ Ok ┊ ┊ ┊ lvm-thick-nvme3n1 ┊ linbit3 ┊ LVM ┊ thickpool-n3 ┊ 27.94 TiB ┊ 27.94 TiB ┊ False ┊ Ok ┊ ┊ ┊ lvm-thick-nvme4n1 ┊ linbit1 ┊ LVM ┊ thickpool-n4 ┊ 27.94 TiB ┊ 27.94 TiB ┊ False ┊ Ok ┊ ┊ ┊ lvm-thick-nvme4n1 ┊ linbit2 ┊ LVM ┊ thickpool-n4 ┊ 27.94 TiB ┊ 27.94 TiB ┊ False ┊ Ok ┊ ┊ ┊ lvm-thick-nvme4n1 ┊ linbit3 ┊ LVM ┊ thickpool-n4 ┊ 27.94 TiB ┊ 27.94 TiB ┊ False ┊ Ok ┊ ┊ ┊ lvm-thick-nvme5n1 ┊ linbit1 ┊ LVM ┊ thickpool-n5 ┊ 27.94 TiB ┊ 27.94 TiB ┊ False ┊ Ok ┊ ┊ ┊ lvm-thick-nvme5n1 ┊ linbit2 ┊ LVM ┊ thickpool-n5 ┊ 27.94 TiB ┊ 27.94 TiB ┊ False ┊ Ok ┊ ┊ ┊ lvm-thick-nvme5n1 ┊ linbit3 ┊ LVM ┊ thickpool-n5 ┊ 27.94 TiB ┊ 27.94 TiB ┊ False ┊ Ok ┊ ┊ ┊ lvm-thick-nvme7n1 ┊ linbit1 ┊ LVM ┊ thickpool-n7 ┊ 27.94 TiB ┊ 27.94 TiB ┊ False ┊ Ok ┊ ┊ ┊ lvm-thick-nvme7n1 ┊ linbit2 ┊ LVM ┊ thickpool-n7 ┊ 27.94 TiB ┊ 27.94 TiB ┊ False ┊ Ok ┊ ┊ ┊ lvm-thick-nvme7n1 ┊ linbit3 ┊ LVM ┊ thickpool-n7 ┊ 27.94 TiB ┊ 27.94 TiB ┊ False ┊ Ok ┊ ┊ ┊ lvm-thick-nvme8n1 ┊ linbit1 ┊ LVM ┊ thickpool-n8 ┊ 27.94 TiB ┊ 27.94 TiB ┊ False ┊ Ok ┊ ┊ ┊ lvm-thick-nvme8n1 ┊ linbit2 ┊ LVM ┊ thickpool-n8 ┊ 27.94 TiB ┊ 27.94 TiB ┊ False ┊ Ok ┊ ┊ ┊ lvm-thick-nvme8n1 ┊ linbit3 ┊ LVM ┊ thickpool-n8 ┊ 27.94 TiB ┊ 27.94 TiB ┊ False ┊ Ok ┊ ┊ ┊ metapool-nvme6n1 ┊ linbit1 ┊ LVM ┊ metapool-n6 ┊ 27.94 TiB ┊ 27.94 TiB ┊ False ┊ Ok ┊ ┊ ┊ metapool-nvme6n1 ┊ linbit2 ┊ LVM ┊ metapool-n6 ┊ 27.94 TiB ┊ 27.94 TiB ┊ False ┊ Ok ┊ ┊ ┊ metapool-nvme6n1 ┊ linbit3 ┊ LVM ┊ metapool-n6 ┊ 27.94 TiB ┊ 27.94 TiB ┊ False ┊ Ok ┊ ┊ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Creating LINSTOR storageClass Create the LINSTOR storageClass manifest and apply it in Kubernetes.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: "linstor-r2-n0"
provisioner: linstor.csi.linbit.com
parameters:
allowRemoteVolumeAccess: "false"
autoPlace: "2"
storagePool: "lvm-thick-nvme0n1"
DrbdOptions/Disk/al-extents: "6007"
DrbdOptions/Disk/disk-barrier: "no"
DrbdOptions/Disk/disk-flushes: "no"
DrbdOptions/Disk/md-flushes: "no"
DrbdOptions/Net/max-buffers: "10000"
property.linstor.csi.linbit.com/StorPoolNameDrbdMeta: "metapool-nvme6n1"
property.linstor.csi.linbit.com/PrefNic: rep_nic
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: "linstor-r2-n1"
provisioner: linstor.csi.linbit.com
parameters:
allowRemoteVolumeAccess: "false"
autoPlace: "2"
storagePool: "lvm-thick-nvme1n1"
DrbdOptions/Disk/al-extents: "6007"
DrbdOptions/Disk/disk-barrier: "no"
DrbdOptions/Disk/disk-flushes: "no"
DrbdOptions/Disk/md-flushes: "no"
DrbdOptions/Net/max-buffers: "10000"
property.linstor.csi.linbit.com/StorPoolNameDrbdMeta: "metapool-nvme6n1"
property.linstor.csi.linbit.com/PrefNic: rep_nic
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: "linstor-r2-n3"
provisioner: linstor.csi.linbit.com
parameters:
allowRemoteVolumeAccess: "false"
autoPlace: "2"
storagePool: "lvm-thick-nvme3n1"
DrbdOptions/Disk/al-extents: "6007"
DrbdOptions/Disk/disk-barrier: "no"
DrbdOptions/Disk/disk-flushes: "no"
DrbdOptions/Disk/md-flushes: "no"
DrbdOptions/Net/max-buffers: "10000"
property.linstor.csi.linbit.com/StorPoolNameDrbdMeta: "metapool-nvme6n1"
property.linstor.csi.linbit.com/PrefNic: rep_nic
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: "linstor-r2-n4"
provisioner: linstor.csi.linbit.com
parameters:
allowRemoteVolumeAccess: "false"
autoPlace: "2"
storagePool: "lvm-thick-nvme4n1"
DrbdOptions/Disk/al-extents: "6007"
DrbdOptions/Disk/disk-barrier: "no"
DrbdOptions/Disk/disk-flushes: "no"
DrbdOptions/Disk/md-flushes: "no"
DrbdOptions/Net/max-buffers: "10000"
property.linstor.csi.linbit.com/StorPoolNameDrbdMeta: "metapool-nvme6n1"
property.linstor.csi.linbit.com/PrefNic: rep_nic
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: "linstor-r2-n5"
provisioner: linstor.csi.linbit.com
parameters:
allowRemoteVolumeAccess: "false"
autoPlace: "2"
storagePool: "lvm-thick-nvme5n1"
DrbdOptions/Disk/al-extents: "6007"
DrbdOptions/Disk/disk-barrier: "no"
DrbdOptions/Disk/disk-flushes: "no"
DrbdOptions/Disk/md-flushes: "no"
DrbdOptions/Net/max-buffers: "10000"
property.linstor.csi.linbit.com/StorPoolNameDrbdMeta: "metapool-nvme6n1"
property.linstor.csi.linbit.com/PrefNic: rep_nic
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: "linstor-r2-n7"
provisioner: linstor.csi.linbit.com
parameters:
allowRemoteVolumeAccess: "false"
autoPlace: "2"
storagePool: "lvm-thick-nvme7n1"
DrbdOptions/Disk/al-extents: "6007"
DrbdOptions/Disk/disk-barrier: "no"
DrbdOptions/Disk/disk-flushes: "no"
DrbdOptions/Disk/md-flushes: "no"
DrbdOptions/Net/max-buffers: "10000"
property.linstor.csi.linbit.com/StorPoolNameDrbdMeta: "metapool-nvme6n1"
property.linstor.csi.linbit.com/PrefNic: rep_nic
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: "linstor-r2-n8"
provisioner: linstor.csi.linbit.com
parameters:
allowRemoteVolumeAccess: "false"
autoPlace: "2"
storagePool: "lvm-thick-nvme8n1"
DrbdOptions/Disk/al-extents: "6007"
DrbdOptions/Disk/disk-barrier: "no"
DrbdOptions/Disk/disk-flushes: "no"
DrbdOptions/Disk/md-flushes: "no"
DrbdOptions/Net/max-buffers: "10000"
property.linstor.csi.linbit.com/StorPoolNameDrbdMeta: "metapool-nvme6n1"
property.linstor.csi.linbit.com/PrefNic: rep_nic
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer ampere@linbit1:~$ microk8s.kubectl apply -f linstor-sc.yaml storageclass.storage.k8s.io/linstor-r2-n0 created storageclass.storage.k8s.io/linstor-r2-n1 created storageclass.storage.k8s.io/linstor-r2-n3 created storageclass.storage.k8s.io/linstor-r2-n4 created storageclass.storage.k8s.io/linstor-r2-n5 created storageclass.storage.k8s.io/linstor-r2-n7 created storageclass.storage.k8s.io/linstor-r2-n8 created
Benchmarking LINBIT SDS - Software Defined Storage
This test procedure uses Kubernetes Jobs and PersistentVolumeClaims (PVCs) provisioned by LINSTOR. All benchmarking Jobs are being run in parallel and individual results from each Job are summarized to calculate the cumulative results.
FIO Container Dockerfile For Jobs
The Dockerfile used to build the container for performance testing is a simple Ubuntu Jammy (22.04) with fio installed:
FROM arm64v8/ubuntu:22.04
LABEL maintainer="kermat60@gmail.com"
RUN apt-get update && apt-get install -y libaio-dev libaio1 fio The image was built and pushed to Docker Hub and is referenced in the YAML manifests as: kermat/arm64-ubuntu- fio:0.0.1.
Jinja2 Templates For Job and PVC Manifests
The Kubernetes Jobs and PVC manifests were rendered from Jinja2 templates using python3.
ampere@linbit1:~$ cat linstor-jobs-r2-block-reads.yaml.j2 \ | python3 -c "from jinja2 import Template; import sys; print(Template(sys.stdin.read()).render());" \ > rendered-jobs-reads.yaml
Note: An initContainer in each Job template was used to poll LINSTOR’s Prometheus monitoring endpoint and calculate the total out of sync (outofsync) blocks across all LINSTOR resources in the cluster in a loop until the result was equal to zero. This ensures that the tests' replicated writes are not competing with initial synchronization traffic on the replication network.
The following are the Jinja2 templates needed for testing:
Random 4KB Read Template
{% set storage_pools = [{ “sp”: “r2-n0” }, { “sp”: “r2-n1” }, { “sp”: “r2-n3” }, { “sp”: “r2-n4” }, { “sp”: “r2-n5” }, { “sp”: “r2-n7” }, { “sp”: “r2-n8” }] -%} {% for p in storage_pools -%} {% set sp = p[“sp”] -%} {% for I in range(6) -%}
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: linstor-{{ sp }}-{{ I }}
spec:
accessModes:
- ReadWriteOnce
volumeMode: Block
resources:
requests:
storage: 10Gi
storageClassName: linstor-{{ sp }}
apiVersion: batch/v1
kind: Job
metadata:
name: fio-bench-{{ sp }}-{{ I }}
labels:
app: linstorbench
namespace: default
spec:
template:
metadata:
labels:
app: linstorbench
spec:
containers:
- name: fio-bench
image: kermat/arm64-ubuntu-fio:0.0.1
command: [“/usr/bin/fio”]
args:
- --name=iops-test
- --filename=/dev/block
- --ioengine=libaio
- --direct=1
- --invalidate=0
- --rw=randread
- --bs=4k
- --runtime=120s
- --numjobs=16
- --iodepth=64
- --group_reporting
- --time_based
- --significant_figures=10
volumeDevices:
- name: linstor-vol
devicePath: /dev/block
securityContext:
capabilities:
add: [“SYS_RAWIO”]
restartPolicy: Never
volumes:
- name: linstor-vol
persistentVolumeClaim:
claimName: linstor-{{ sp }}-{{ I }}
initContainers:
- name: wait-for-sync
image: curlimages/curl
command: [“/bin/sh”]
args: [“-c”, “sleep 10s; t=1; while [ $t -gt 0 ]; do t=0; sleep 2s; for I in $(curl -s
linstor-op-ns-node-monitoring.default.svc:9942 | grep outofsync | cut -d’ ‘ -f2 | grep -o [0-
9]*); do t=$(($t+$i)); done; done”]
backoffLimit: 4 {% endfor -%} {% endfor -%}
Random 4KB Write Template
{% set storage_pools = [{ “sp”: “r2-n0” }, { “sp”: “r2-n1” }, { “sp”: “r2-n3” }, { “sp”: “r2-n4” }, { “sp”: “r2-n5” }, { “sp”: “r2-n7” }, { “sp”: “r2-n8” }] -%} {% for p in storage_pools -%} {% set sp = p[“sp”] -%} {% for I in range(6) -%}
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: linstor-{{ sp }}-{{ I }}
spec:
accessModes:
- ReadWriteOnce
volumeMode: Block
resources:
requests:
storage: 10Gi
storageClassName: linstor-{{ sp }}
apiVersion: batch/v1
kind: Job
metadata:
name: fio-bench-{{ sp }}-{{ I }}
labels:
app: linstorbench
namespace: default
spec:
template:
metadata:
labels:
app: linstorbench
spec:
containers:
- name: fio-bench
image: kermat/arm64-ubuntu-fio:0.0.1
command: [“/usr/bin/fio”]
args:
- --name=iops-test
- --filename=/dev/block
- --ioengine=libaio
- --direct=1
- --invalidate=0
- --rw=randwrite
- --bs=4k
- --runtime=120s
- --numjobs=16
- --iodepth=64
- --group_reporting
- --time_based
- --significant_figures=10
volumeDevices:
- name: linstor-vol
devicePath: /dev/block
securityContext:
capabilities:
add: [“SYS_RAWIO”]
restartPolicy: Never
volumes:
- name: linstor-vol
persistentVolumeClaim:
claimName: linstor-{{ sp }}-{{ I }}
initContainers:
- name: wait-for-sync
image: curlimages/curl
command: [“/bin/sh”]
args: [“-c”, “sleep 10s; t=1; while [ $t -gt 0 ]; do t=0; sleep 2s; for I in $(curl -s
linstor-op-ns-node-monitoring.default.svc:9942 | grep outofsync | cut -d’ ‘ -f2 | grep -o [0-
9]*); do t=$(($t+$i)); done; done”]
backoffLimit: 4 {% endfor -%} {% endfor -%}
Random 4KB Mixed Read and Write (70/30) Template
{% set storage_pools = [{ “sp”: “r2-n0” }, { “sp”: “r2-n1” }, { “sp”: “r2-n3” }, { “sp”: “r2-n4” }, { “sp”: “r2-n5” }, { “sp”: “r2-n7” }, { “sp”: “r2-n8” }] -%} {% for p in storage_pools -%} {% set sp = p[“sp”] -%} {% for I in range(6) -%}
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: linstor-{{ sp }}-{{ I }}
spec:
accessModes:
- ReadWriteOnce
volumeMode: Block
resources:
requests:
storage: 10Gi
storageClassName: linstor-{{ sp }}
apiVersion: batch/v1
kind: Job
metadata:
name: fio-bench-{{ sp }}-{{ I }}
labels:
app: linstorbench
namespace: default
spec:
template:
metadata:
labels:
app: linstorbench
spec:
containers:
- name: fio-bench
image: kermat/arm64-ubuntu-fio:0.0.1
command: [“/usr/bin/fio”]
args:
- --name=iops-test
- --filename=/dev/block
- --ioengine=libaio
- --direct=1
- --invalidate=0
- --rw=randrw
- --rwmixread=70
- --bs=4k
- --runtime=120s
- --numjobs=16
- --iodepth=64
- --group_reporting
- --time_based
- --significant_figures=10
volumeDevices:
- name: linstor-vol
devicePath: /dev/block
securityContext:
capabilities:
add: [“SYS_RAWIO”]
restartPolicy: Never
volumes:
- name: linstor-vol
persistentVolumeClaim:
claimName: linstor-{{ sp }}-{{ I }}
initContainers:
- name: wait-for-sync
image: curlimages/curl
command: [“/bin/sh”]
args: [“-c”, “sleep 10s; t=1; while [ $t -gt 0 ]; do t=0; sleep 2s; for I in $(curl -s
linstor-op-ns-node-monitoring.default.svc:9942 | grep outofsync | cut -d’ ‘ -f2 | grep -o [0-
9]*); do t=$(($t+$i)); done; done”]
backoffLimit: 4 {% endfor -%} {% endfor -%}
Sequential 128KB Read Template
{% set storage_pools = [{ “sp”: “r2-n0” }, { “sp”: “r2-n1” }, { “sp”: “r2-n3” }, { “sp”: “r2-n4” }, { “sp”: “r2-n5” }, { “sp”: “r2-n7” }, { “sp”: “r2-n8” }] -%} {% for p in storage_pools -%} {% set sp = p[“sp”] -%} {% for I in range(6) -%}
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: linstor-{{ sp }}-{{ I }}
spec:
accessModes:
- ReadWriteOnce
volumeMode: Block
resources:
requests:
storage: 10Gi
storageClassName: linstor-{{ sp }}
apiVersion: batch/v1
kind: Job
metadata:
name: fio-bench-{{ sp }}-{{ I }}
labels:
app: linstorbench
namespace: default
spec:
template:
metadata:
labels:
app: linstorbench
spec:
containers:
- name: fio-bench
image: kermat/arm64-ubuntu-fio:0.0.1
command: [“/usr/bin/fio”]
args:
- --name=iops-test
- --filename=/dev/block
- --ioengine=libaio
- --direct=1
- --invalidate=0
- --rw=read
- --bs=128k
- --runtime=120s
- --numjobs=4
- --iodepth=64
- --group_reporting
- --time_based
- --significant_figures=10
volumeDevices:
- name: linstor-vol
devicePath: /dev/block
securityContext:
capabilities:
add: [“SYS_RAWIO”]
restartPolicy: Never
volumes:
- name: linstor-vol
persistentVolumeClaim:
claimName: linstor-{{ sp }}-{{ I }}
initContainers:
- name: wait-for-sync
image: curlimages/curl
command: [“/bin/sh”]
args: [“-c”, “sleep 10s; t=1; while [ $t -gt 0 ]; do t=0; sleep 2s; for I in $(curl -s
linstor-op-ns-node-monitoring.default.svc:9942 | grep outofsync | cut -d’ ‘ -f2 | grep -o [0-
9]*); do t=$(($t+$i)); done; done”]
backoffLimit: 4 {% endfor -%} {% endfor -%}
Sequential 128KB Write Template
{% set storage_pools = [{ “sp”: “r2-n0” }, { “sp”: “r2-n1” }, { “sp”: “r2-n3” }, { “sp”: “r2-n4” }, { “sp”: “r2-n5” }, { “sp”: “r2-n7” }, { “sp”: “r2-n8” }] -%} {% for p in storage_pools -%} {% set sp = p[“sp”] -%} {% for I in range(6) -%}
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: linstor-{{ sp }}-{{ I }}
spec:
accessModes:
- ReadWriteOnce
volumeMode: Block
resources:
requests:
storage: 10Gi
storageClassName: linstor-{{ sp }}
apiVersion: batch/v1
kind: Job
metadata:
name: fio-bench-{{ sp }}-{{ I }}
labels:
app: linstorbench
namespace: default
spec:
template:
metadata:
labels:
app: linstorbench
spec:
containers:
- name: fio-bench
image: kermat/arm64-ubuntu-fio:0.0.1
command: [“/usr/bin/fio”]
args:
- --name=iops-test
- --filename=/dev/block
- --ioengine=libaio
- --direct=1
- --invalidate=0
- --rw=write
- --bs=128k
- --runtime=120s
- --numjobs=4
- --iodepth=64
- --group_reporting
- --time_based
- --significant_figures=10
volumeDevices:
- name: linstor-vol
devicePath: /dev/block
securityContext:
capabilities:
add: [“SYS_RAWIO”]
restartPolicy: Never
volumes:
- name: linstor-vol
persistentVolumeClaim:
claimName: linstor-{{ sp }}-{{ I }}
initContainers:
- name: wait-for-sync
image: curlimages/curl
command: [“/bin/sh”]
args: [“-c”, “sleep 10s; t=1; while [ $t -gt 0 ]; do t=0; sleep 2s; for I in $(curl -s
linstor-op-ns-node-monitoring.default.svc:9942 | grep outofsync | cut -d’ ‘ -f2 | grep -o [0-
9]*); do t=$(($t+$i)); done; done"]
backoffLimit: 4 {% endfor -%} {% endfor -%}
Running Benchmarking and Calculating Results
With the Jinja2 templates rendered as YAML manifests, each individual test is run by applying the test’s respective manifest:
ampere@linbit1:~$ microk8s.kubectl apply -f rendered-jobs-reads.yaml
When the Jobs created by applying the test manifests are completed, the results are saved by retrieving the log from all Jobs and redirecting them into a single log file for each test:
ampere@linbit1:~$ microk8s.kubectl logs -c fio-bench -l app=linstorbench --tail=-1 > test-results/randread-logs.txt
Results are calculated from the saved logs using the following bash commands.
For calculating IOPS:
ampere@linbit1:~$ total=0; \ for iops in $(grep -o "IOPS=[0-9].*\," test-results/randread-logs.txt | sed 's/IOPS=//g' | sed 's/\,//g'); \ do total=$((total + iops)); done; echo $total
For calculating bandwidth:
ampere@linbit1:~$ total=0; \ for bw in $(grep -o "BW=[0-9]*B\/s" test-results/seqwrite-logs.txt | sed 's/BW=//g' | sed 's/B\/s//g'); \ do total=$((total + bw)); done; echo $total
This concludes the tutorial for setting up a LINBIT SDS MicroK8s cluster on Ampere Altra processors. For further questions on how to realize the performance gains and power savings of Ampere’s revolutionary processors, please contact us at https://amperecomputing.com/company/contact-sales
Additional Information and Resources
LINBIT SDS Brief on Ampere Solution Domain.
Ampere Software Defined Storage Solution Page.
Benchmarking on Ampere Altra Max Platform with LINBIT SDS on LINBIT website.
Join LINBIT’s GitHub Organization.
Joint LINBIT’s Community on Slack.
The DRBD® and LINDTOR® User’s Guide.
drbd-announce: Announcements of new releases and critical bugs found.
Appendix A: Legalese
Ampere Computing reserves the right to make changes to its products, its datasheets, or related documentation, without notice and warrants its products solely pursuant to its terms and conditions of sale, only to substantially comply with the latest available datasheet. Ampere, Ampere Computing, the Ampere Computing and ‘A’ logos, and Altra are registered trademarks of Ampere Computing.
Arm is a registered trademark of Arm Limited (or its subsidiaries) in the US and/or elsewhere. All other trademarks are the property of their respective holders.
LINBIT®, the LINBIT logo, DRBD®, the DRBD logo, LINSTOR®, and the LINSTOR logo are trademarks or registered trademarks of LINBIT in Austria, the EU, the United States, and many other countries. Other names mentioned in this document may be trademarks or registered trademarks of their respective owners.
Ampere Computing LLC
4655 Great America Parkway Suite 601
Santa Clara, CA 95054
