The Ampere® PMU Profiler is a python-tool designed to give developers and performance engineers deep insight into the microarchitectural behavior of applications running on Ampere CPUs. While standard profilers identify where time is spent (e.g., in which function), the Ampere PMU Profiler explains why that time is being spent by measuring low-level hardware events.

This capability is crucial for unlocking the full performance potential of Ampere CPUs. By understanding CPU-level bottlenecks such as inefficient instruction fetching, data cache misses, or branch mispredictions engineers can make targeted code optimizations that are impossible to identify with application-level tools alone. This document outlines a top-down performance analysis methodology, positioning the Ampere PMU Profiler as the microarchitecture-focused stage of that workflow, and provides a practical guide for its use. After broad system triage, APP explains whether a workload is frontend bound, backend bound, memory limited, or coherence limited, and that guidance can then be paired with hotspot tools for function and loop-level attribution. Adopting this tool will lead to more efficient software, better resource utilization, and a stronger competitive advantage for workloads running on Ampere processors.

The Ampere PMU Profiler is a Linux-based python utility that provides a user-friendly interface to the Performance Monitoring Unit (PMU) on Ampere CPUs (e.g., Ampere® Altra®, AmpereOne®)

• What is a PMU? A PMU is a specialized hardware component within a CPU. It contains a set of counters that can be programmed to count specific hardware events. These events represent the fundamental operations and potential inefficiencies of the CPU core, such as:
  • Instructions retired
  • CPU cycles
  • Cache misses (L1, L2, L3)
  • Branch mispredictions
  • Stalls in the instruction pipeline

The Ampere PMU Profiler leverages the underlying Linux perf subsystem but simplifies the process by providing a curated list of Ampere-specific event names, pre-defined event groups for common analysis scenarios, and a streamlined workflow. This is implemented in python and is open sourced at https://github.com/AmpereComputing/ampere-pmu-profiler

In a competitive landscape, simply running software is not enough; it must run efficiently. The Ampere PMU Profiler is essential for achieving this goal for several key reasons:

• To Answer "Why" is My Code Slow?
  • Application Profilers (gprofng, Valgrind): Tell you which function is slow.
  • APP: Tells you why the hardware is executing that function slowly. Is it waiting for memory? Is it fetching instructions inefficiently? This is the missing piece of the puzzle.
• Unlocking the Full Potential of Hardware: Ampere CPUs are designed for high-throughput, cloud-native workloads with a high core count. Generic code may not automatically take full advantage of our specific microarchitectural strengths. The PMU profiler provides the data needed to optimize algorithms and data structures specifically for our hardware.
• Data-Driven Optimization: It moves performance tuning from guesswork ("I think it's a memory issue") to a scientific, data-driven process ("The L2 data cache miss rate is 15%, which is correlated with a pipeline stall of 30%"). This saves engineering time and leads to more effective optimizations.
• Identifying "Invisible" Bottlenecks: Some of the most significant performance issues are not obvious from source code. For example, false sharing in multi-threaded applications, poor instruction cache layout, or DRAM bank conflicts are only visible through hardware performance counters.
• Competitive Benchmarking and Analysis: When analyzing our performance against competitors or evaluating a new software stack, the PMU Profiler allows us to understand performance differences at the most fundamental level.

The Ampere PMU Profiler is usually not the very first tool in an investigation, but once the workload is known to be CPU-bound or is showing an unexplained regression or scaling issue, it can be brought in early. In practice it complements hotspot tools rather than sitting strictly before or after them. It should be used:

• During CPU-bound triage or after hotspots are identified: Use APP to get a workload-wide microarchitectural view (frontend vs. backend, memory vs. core, coherence vs. capacity), then pair it with perf record, perf report, or flame graphs when you need function- or loop-level attribution.
• When performance scaling is sub-linear: When adding more threads or cores doesn't result in the expected performance improvement.
• To investigate performance regressions: When a code change results in a slowdown that isn't easily explained by the code logic itself.
• For final-stage performance tuning: To squeeze the last 10-20% of performance out of a critical application by fine-tuning its interaction with the hardware.
• To evaluate compiler effectiveness: To check if compiler optimizations (like vectorization or loop unrolling) are having the desired effect on the hardware.

app -n 120 -c 1,2 -i 1 --tda -o <my_folder> -j "taskset -c 1,2 "

the above example command collects

• PMU profiling samples for 120s
• With a sampling interval of 1s
• Collects profiles for cores 1 and 2
• TopDown metrics and render a TDA sunburst chart
• PMU profiles while running the workload affinitized to cores 1 and 2.

• Goal: Understand the overall system health and identify the primary resource bottleneck. Is the application limited by CPU, Memory, Disk I/O, or Network?
• Key Questions:
  • Is the overall CPU utilization high?
  • Is the system swapping or under memory pressure?
  • Is the application spending a lot of time waiting for I/O (iowait)?
• Common Tools: top, htop, vmstat, sar, iostat, netstat and Ampere System Profiler
• Outcome: A high-level characterization of the bottleneck. If the workload is confirmed to be CPU-bound (high user-space CPU utilization), you proceed to the next level.

• Goal: Identify the "hotspots" within the application. Which functions or code paths are consuming the most CPU time?
• Key Questions:
  • Where is my application spending time?
  • What is the call graph hierarchy leading to these hotspots?
• Common Tools: perf record/perf report, gprof, Valgrind/callgrind.
• Outcome: A list of one or more functions that are the primary targets for optimization. You now know where to look, but not yet why it's slow.

• Goal: Understand why the identified "hot" functions are slow by analyzing their interaction with the CPU hardware.
• Key Concepts:
  • Frontend vs. Backend: The CPU pipeline is broadly split. The Frontend fetches instructions, decodes them, and predicts branches. The Backend executes the instructions (ALU operations, memory access). A stall in one could starve the other.
  • Instructions Per Cycle (IPC): A key measure of efficiency. IPC = (Instructions Retired) / (CPU Cycles). An IPC below 1.0 often indicates stalls. A high IPC (e.g., >2.0) indicates high efficiency.
  • Memory Hierarchy: Accessing data gets exponentially slower as you move from registers -> L1 cache -> L2 cache -> L3 cache -> DRAM. Cache misses are a primary cause of performance loss.
• The Ampere PMU Profiler's Role: At this level, APP can be used to collect hardware events related to microarchitecture level analysis as described in Figure 1 to understand where stalls might be in your pipeline.

Analysis Workflow:

• Form a Hypothesis: Use APP's workload-wide signature together with hotspot tools to identify the function or loop you want to explain. APP tells you what kind of bottleneck you have; perf record or flame graphs tell you where it occurs.
• Measure a Baseline with Ampere PMU Profiler: Capture APP data for the baseline workload using fixed input, core affinity, runtime length, and system settings.
• Change One Thing and Re-run: Modify the target loop, data layout, or synchronization pattern and rerun the same workload under identical conditions
  • Compare Before vs. After: Use the HTML report and TDA sunburst to verify that the suspected bottleneck moves in the expected direction, for example higher IPC, lower backend-memory stalls, lower cache or TLB MPKI, or fewer coherence events.
  • Start with the dominant TDA domain: Identify the largest top-level category first before drilling into subdomains.
  • Drill down hierarchically: Follow only the flagged subdomains and ignore low-signal branches until the remaining candidates are specific enough to act on.
  • Corroborate with broader APP metrics: Check cache, TLB, memory-bandwidth, coherence, and mesh or uncore metrics under the same test conditions.
• Close the loop with code attribution when needed: If the microarchitectural diagnosis is clear but the exact loop is not, return to perf record, perf report, flame graphs, or source inspection to map the signal back to the precise code change.

To illustrate how the Ampere PMU Profiler and TopDown Accounting (TDA) helped identify performance limits in real HPC codes, we profiled DGEMM (double‑precision GEMM) from BLIS, a canonical dense linear algebra kernel. DGEMM is expected to scale with matrix dimension until it becomes limited by core compute throughput and/or microarchitectural resources, rather than by memory bandwidth.

Across increasing matrix sizes, DGEMM scales well and plateaus at ~3.6 TFLOPS (Figure 2), consistent with a compute-dominated steady state where additional data reuse and blocking reduce sensitivity to external memory. During each matrix multiplication iteration, CPU utilization peaks near 100%. APP enables us to collect DRAM bandwidth for both read and write and these DRAM metrics further support a compute-bound interpretation. On this system, the maximum sustained bandwidth measured by STREAM Triad is ~252 GB/s, while DGEMM reaches a peak of ~117 GB/s (47% of peak). This substantial bandwidth headroom shows that DGEMM performance is not constrained by DRAM bandwidth; instead, the dominant limits are inside the core pipeline and execution resources

Microarchitectural Bottleneck Diagnosis via TDA:

The Ampere PMU Profiler’s Top‑Down Accounting (TDA) (Figure 3) breaks down where cycles are spent and why throughput saturates:

• Level 1 - (Pipeline): Backend Bound dominates DGEMM is primarily backend bound, meaning the frontend is generally able to supply work, and performance is limited by the backend’s ability to execute/complete instructions.
• Level 2 - Core Bound: The backend time is largely core bound, indicating that cycles are spent actively executing compute (dense FP math) rather than waiting on the memory hierarchy. This aligns with DGEMM’s expected behavior once blocked kernels operate from cache.
• Level 3 - Resource Bound: (Execution Unit Pressure, especially FSU) Within core bound, TDA attributes the dominant component to resource bound, highlighting pressure on execution resources most notably the FSU (floating‑point/SIMD units). This suggests DGEMM is nearing the sustainable throughput limits of the FP/vector pipelines (e.g., issue/execute bandwidth, unit availability, or related scheduler/queueing constraints) rather than stalling on data delivery.
• Low “L2/Memory” contribution: The relatively small portion attributed to memory at Level 2 reinforces that DGEMM is not memory bandwidth bound on AmpereOne for the profiled sizes and implementation.

A common multithreaded symptom is poor scaling even when each thread updates a logically independent variable. False sharing occurs when those variables live on the same cache line, so each write invalidates another core's copy and turns independent updates into coherence traffic.

Method: Using APP to distinguish coherence ping-pong from generic memory pressure

We started from that symptom and used APP to compare the slow layout against a padded or aligned control under identical CPU pinning. The diagnostic question was whether the slowdown came from cache capacity or DRAM bandwidth limits, or from coherency traffic caused by cache-line bouncing.

• IPC drops from 1.17 in the aligned control to 0.47 in the suspected-sharing case, a roughly 2.5x reduction in useful work per cycle.
• l2d_inv_pki jumps from 0.01 to 6.74. That is a coherence signature: the cache hierarchy is repeatedly invalidating lines instead of simply serving steady-state hits and misses.
• l2_snoops_pki rises from 0.06 to 6.85, showing the interconnect is flooded with coherence probes and ownership transfers rather than ordinary memory traffic.
• l2d_inv_per_snoop rises from 0.15 to 0.99. In other words, nearly every snoop in the slow case forces an invalidation, which is what you expect when two cores keep contending for write permission on the same cache line.

Taken together, these results show how APP can detect false sharing from a coherence signature. The combination of lower IPC, sharply higher invalidation and snoop activity, and recovery in the padded or aligned control points to cache-line ping-pong rather than generic memory pressure. In other words, APP was able to identify false sharing as the cause of the slowdown

We profiled Meta’s LLaMA benchmark using the llama.cpp inference engine with a Llama 3.1 8B quantized (int8) model to understand the limitations of CPU-side inference throughput. The key observation was that the workload behaves differently across phases: prompt processing is relatively compute heavy, whereas token generation becomes much more memory intensive because model weights and KV-cache data must be repeatedly moved through the memory hierarchy. For that reason, the analysis focused on the steady-state token generation phase rather than mixing it with prompt processing.

Method: Using phase-aware TDA and PMU metrics to separate compute limits from memory limits

We verified that the workload was spending more than 90% of CPU time in user space, which ruled out obvious kernel or system-level overheads. APP was used to collect TopDown Accounting (TDA) on the active inference core and PMU events over 10-second windows, timing collections separately for prompt processing and token generation. That phase separation mattered, because combining the two would blur a compute-bound region with a memory-bound one.

• TDA showed the token-generation phase to be highly backend bound at peak load, which immediately pointed away from a frontend delivery problem and toward data-side stalls.
• As the number of inference processes increased, token-generation throughput scaled near-proportionally up to three processes. Beyond that point, memory bandwidth saturated while useful throughput plateaued, showing that additional parallelism was no longer translating into more tokens per second.
• In the code-level deep dive, the IPC of the top function showed long low-IPC troughs alternating with higher-IPC regions. Correlating those periods with APP metrics and perf-guided hotspot analysis helped narrow the bottleneck and reduce the frequency and duration of those stalls, yielding more than 2x performance improvement.

This case study shows how APP and TopDown Accounting (TDA) can clearly expose the real bottleneck in LLM inference. By isolating the token-generation phase, APP showed that performance was primarily limited by backend stalls driven by memory-bandwidth pressure.