Hadoop Workload Brief
Hadoop is open-source framework to store and process large data sets from gigabytes to petabytes. Hadoop is designed to scale up from a single computer to thousands, each offering local computation and storage.
Ampere—Empowering What’s Next
Ampere® Altra® processors are a complete system-on-chip (SOC) solution built for Cloud Native applications. Ampere Altra supports up to 80 aarch64 cores. In addition to delivering a large number of high-performance cores, its innovative architecture delivers predictable high performance, linear scaling, and power efficiency.
Apache Hadoop framework is designed for distributed processing of large data sets. Hadoop is designed to scale out from a single server to thousands of machines, each offering local computation, storage, or both. When implemented in a cluster, the software has a built-in resiliency to handle a failed server or a failed component in a server.
Hadoop is an open-source framework to store and process large data sets from gigabytes to petabytes. Hadoop clusters multiple computers to analyze data in parallel. It consists of four main modules, HDFS (Hadoop Distributed File System), YARN (Yet Another Resource Negotiator), Map Reduce (MapR) and Hadoop Common. Applications collect data in various formats and seed it to the cluster. The name node, that is the center piece of HDFS file system has metadata information of all chunks of data and keeps the directory tree of all files in the file system and tracks where across the cluster the file data is kept. A MapR job runs against this data in HDFS across data nodes.
All the above tasks are computationally intensive. The data must be pulled from HDFS, which demands high-performance storage, must be coordinated across different computers, that demand high-speed networks and must be quickly processed by thousands of tasks and finally aggregated by reducers to organize the final output.
Hadoop on Ampere Altra processors
Ampere technology packs more cores per socket, maximizing the number of cores per rack. Clusters using Ampere Altra processors benefit from a power optimized design, enabling lower power consumption and predictable performance for big data applications and other data lake technologies. Higher core density per rack increases the rack efficiency, enabled by lower power and cooling requirements, which drives Capex and Opex savings.
In this solution brief, we walk you through testing completed with TPCx-HS and Hadoop TeraSort performance benchmarks executed on a 9-node cluster each with an Altra Q80-30 processor for a total of 720 cores in the rack.
Benefits of Running Hadoop on Ampere Altra
Cloud Native: Designed from the ground up for “born in the cloud” workloads, Ampere Altra can deliver much higher performance over its x86 peers.
Consistency and Predictability: Ampere Altra processors that are designed for cloud native usage, provide consistent and predictable performance of Hadoop solutions and bursting workloads.
Scalable: With an innovative scale-out architecture, Ampere Altra processors have a high core count with compelling single-threaded performance. Combined with consistent frequencies for all cores, Ampere processors make big data workloads scale up and scale out efficiently.
Power Efficient: Industry-leading energy efficiency allows Ampere Altra processors to hit competitive levels of raw performance while consuming much lower power than the competition.
Ampere Altra
- 80 64-bit CPU cores up to 3.30 GHz
- 64 KB L1 I-cache, 64 KB L1 D-cache per core
- 1 MB L2 cache per core
- 32 MB System Level Cache (SLC)
- 2x full-width (128b) SIMD
- Coherent mesh-based interconnect
Memory
- 8x 72-bit DDR4-3200 channels
- ECC and DDR4 RAS
- Up to 16 DIMMs and 4 TB addressable memory
Connectivity
- 128 lanes of PCIe Gen4
- Coherent multi-socket support
- 4 x16 CCIX lanes
Technology & Functionality
- Arm v8.2+, SBSA Level 4
- Advanced Power Management
Performance
- SPECrate®2017 Integer Estimated: 300
Hadoop Architecture
Benchmarking Configuration
We used both HiBench and TPCx-HS benchmarking tools to evaluate the performance of a nine node Hadoop cluster.
TPCx-HS benchmark is an industry standard benchmark designed to stress both hardware and software that is based on Apache HDFS compatible distributions. It is used to assess a broad range of system topologies and implementation methodologies in big data systems.
HiBench is a big data benchmark suite that helps evaluate different big data frameworks in terms of speed, throughput, and system resource utilization. HiBench was used to measure the TeraSort output of the cluster scaling from 1 to 9 nodes.
The benchmarks were first run on a single node followed by adding more data nodes to the cluster. The first node functioned both as name node and data node. Hadoop configuration parameters were tuned to maximize the CPU, storage and network utilization to boost the throughput of the cluster.
Details of the configuration can be found here. here
Benchmarking Results
Hadoop Benchmark on a 9-Node Bare Metal Cluster
-
Both the benchmark tools drove Cluster CPU utilization above 80%.
-
The NVMe storage disk utilization was observed around 90% on all the data nodes.
-
Linear scalability observed with both TPCx-HS and HiBench tools in a scale out architecture.
-
Ampere servers CPU power consumption, monitored with lm-sensors, was between 120-140 watts of usage power per server during the benchmark run.
Benchmarking Conclusions
Ampere Altra processors provide exceptional power, linear scalability, and high performance per node in a scale-out cluster. Solutions like big data require massive computational power and persistent storage. Ampere Altra processors scale up linearly with workloads while Hadoop and MapReduce frameworks benefit from the linear scale out architecture. Running big data applications on Ampere Altra processors harnesses both the scale up and scale out architectures. Densely packed cores in Ampere architecture reduce DC space while the low power consumption will reduce total power consumption and cooling requirements of any Hadoop Cluster, providing a better return of Investment.
Footnotes
All data and information contained herein is for informational purposes only and Ampere reserves the right to change it without notice. This document may contain technical inaccuracies, omissions and typographical errors, and Ampere is under no obligation to update or correct this information. Ampere makes no representations or warranties of any kind, including but not limited to express or implied guarantees of noninfringement, merchantability, or fitness for a particular purpose, and assumes no liability of any kind. All information is provided “AS IS.” This document is not an offer or a binding commitment by Ampere. Use of the products contemplated herein requires the subsequent negotiation and execution of a definitive agreement or is subject to Ampere’s Terms and Conditions for the Sale of Goods.
©2022 Ampere Computing. All Rights Reserved. Ampere, Ampere Computing, Altra and the ‘A’ logo are all registered trademarks or trademarks of Ampere Computing. Arm is a registered trademark of Arm Limited (or its subsidiaries). All other product names used in this publication are for identification purposes only and may be trademarks of their respective companies.
Ampere Computing® / 4655 Great America Parkway, Suite 601 / Santa Clara, CA 95054 / amperecomputing.com
