Cluster Configuration

System - 9x single socket Ampere Servers with 80 cores and 3.0 GHz

Memory - 16 x 32GB DDR4 DIMMS per node

Network - Mallanox CX6 @ 100 Gbps

OS - Ubuntu 20.04 5.4 Kernal

Hadoop Configuration

dfs.block.size - 512M

yarn.scheduler.minimum-allocation-mb - 2048

yarn.scheduler.maximum-allocation-mb - 69632

yarn.scheduler.maximum-allocation-vcores - 78

mapreduce.reduce.parallel.copies - 64

mapreduce.reduce.shuffle.parallelcopies - 64

mapreduce.map.java.opts - 2048M