The Ampere® PMU Profiler (APP) is a Python-based tool designed to provide deep insight into the microarchitectural behavior of applications running on Ampere CPUs (e.g., Ampere® Altra® and AmpereOne®). Unlike standard profilers that identify where time is spent (for example, which functions consume CPU time), the PMU Profiler explains why time is being spent by measuring low-level hardware events associated with the CPU pipeline and execution behavior.

A key outcome of APP is that it enables performance engineers to move from coarse symptoms to actionable causes. For example, while application-level profiling can show an expensive code path, APP can help identify whether the expense stems from inefficient instruction fetching, data cache misses, or other microarchitectural factors that are difficult or impossible to isolate using application-level tools alone. The document outlines a top-down performance analysis methodology and positions APP as an essential final step for expert-level tuning, particularly on Ampere platforms, where you must understand hardware-level bottlenecks and then apply targeted code optimizations.

APP is intended to complement system-level analysis rather than replace it. System-level profilers are useful for identifying high-level bottlenecks such as resource saturation or contention, but APP is focused on microarchitecture-level analysis by collecting hardware events. This makes APP especially valuable after system bottlenecks have been eliminated or ruled out, leaving “microarchitecture inefficiency” as the remaining likely cause of slowdowns

The Ampere PMU Profiler uses Linux perf utility with validated pmus and metrics on Ampere CPUs. Its purpose is to capture microarchitectural performance indicators through hardware event measurement. In practice, this means APP collects events measured by perf stat that relate to the CPU pipeline and execution mechanisms, allowing engineers to determine not only what is slow, but the underlying microarchitectural reason.

A central distinction between APP and application profiling tools is the level of visibility. Tools that sample stack traces (or count function invocations) typically answer: “Which functions are active during the slow period?” APP answers a more hardware-specific question: “Which microarchitectural mechanisms are consuming cycles, and what stalls or inefficiencies are present?”

The APP workflow assumes that developers can form a hypothesis about where the bottleneck likely originates, such as a particular loop or data access pattern and then rely on PMU event measurements to confirm or refute those hypotheses at the microarchitectural level

Performance problems are frequently multi-layered. Even after system-level bottlenecks are addressed (for example, ensuring that CPU is not idling due to I/O, ensuring there is sufficient memory, and verifying resource utilization), some workloads still perform poorly because the CPU spends cycles in inefficient pipeline states.

APP helps solve this class of problems by measuring hardware-level behavior. For example, APP can identify microarchitectural bottlenecks such as:

• inefficient instruction fetching,
• data cache misses,
• branch-related pipeline effects,
• and other pipeline-level stall sources that manifest as lost cycles.

This capability is important because microarchitectural causes often do not map cleanly to application symptoms. Code can appear “hot” in a profiler, but the reason it is slow might be due to how it interacts with cache hierarchies, how it causes translation or fetch inefficiencies, or how the processor recovers from pipeline disruptions. Those details are what PMU-based measurement aims to expose.

APP also links investigation to “unlocking the full performance potential” of the hardware. By understanding CPU-level bottlenecks, engineers can choose targeted optimizations that application-level tools alone cannot determine with confidence. This ultimately leads to more efficient software and better utilization of Ampere hardware for competitive workloads

Performance tuning is a process of systematic investigation, moving from a broad, system-wide view down to the specific interactions between code and hardware. 

Performance Optimization is as much art as it is science. The APEX framework uses tools and methodologies to add structure and rigor to the process and can bridge the gap between creative intuition and empirical fact.

We propose applying the APEX (Adaptive Profiling and Execution) methodology to enable root cause analysis for solving performance problems. Follow the funnel above from top to bottom to effectively use the procedure.

The methodology recommends starting with assessing platform health as a first step to ensure that platform used for performance analysis is set up well. An unhealthy platform may mislead the performance analysis.

Consider capturing initial performance metrics before tuning any system or application settings. This establishes a clear understanding of the current workload and identifies key scalability knobs. We recommend using Ampere’s PerfKit Benchmarker (APB), which supports many open-source applications, to create a reliable baseline for further analysis and tuning.

Next, is to assess system performance and any hardware or system bottlenecks while the code is – this is where Ampere System Profiler (ASP) is useful to eliminate any system or resource bottlenecks. ASP can also be used to right-size the instance shape and ensure that the compute resources are efficiently consumed by the workload.

One method which may be used is to leverage APB’s automated benchmarking framework to start and stop ASP’s collectors during run phase of a given APB benchmark. This ensures that profile is collected while critical code paths are executed and clear report profile is generated.

Once system and resource bottlenecks are eliminated, if the performance issue persists and points to CPUcycles not being used efficiently, we propose going to the next step in the pyramid and use Ampere PMU Profiler to root-cause the issue further.

Finally, system benchmarking should be done after all bottlenecks are resolved or analyzed to effectively measure the system’s performance for the workload.

Following this systematic APEX methodology ensures that we eliminate possible issues as a part of a structured process to efficiently conduct root cause analysis.[

At the microarchitecture level, performance is shaped by how the CPU pipeline handles instruction delivery, execution, and memory access. APP leverages PMU measurements to identify pipeline behavior and stall sources.

APP emphasizes the performance significance of the memory hierarchy. As data access moves from registers to L1 cache, then L2 cache, then L3 cache, and finally DRAM, access becomes exponentially slower. Because of this, cache misses are a primary cause of performance loss. This provides a conceptual foundation for many APP investigations: if a workload touches large working sets or accesses data in a non-contiguous pattern, it may trigger cache misses that increase effective latency and reduce throughput.

APP can be used at the microarchitecture level to understand where stalls might be in the pipeline. The APP role is to collect hardware events related to pipeline stall behavior and to use those events to characterize the workload’s execution profile. This capability matters because pipeline stalls and inefficiencies can dominate runtime even when application-level profiling points to a “hot” function without explaining the root cause.

APP is structured around answering questions that cannot be fully resolved with application-level profiling alone. Based on the described APP workflow and report interpretation strategy, APP can help you answer:

1. Where are cycles going at the microarchitectural level? The APP HTML report and TDA sunburst charts are used to broadly characterize whether time is dominated by categories such as instruction retirement behavior, frontend bound behavior, or backend bound behavior.

2. Which stall or inefficiency class is consistent with the hot code path? Once you hypothesize a bottleneck mechanism (e.g., cache misses from non-contiguous access), APP measurements can confirm whether the observed behavior aligns with that mechanism.

3. What microarchitectural reason explains a hot function’s cost? APP’s purpose is explicitly to explain why time is spent by measuring hardware events. This allows developers to translate hot functions into hardware interactions that can be optimized.

4. Is the workload limited by instruction delivery versus execution/memory? By inspecting broad characterization categories (frontend vs backend bound) in the APP HTML report, engineers can determine which side of the pipeline is more likely to be limiting performance.

Copy
app -n 120 -c 1,2 -i 1 –tda -o <folder> -j “taskset -c1,2 <workload>

The above example command attempts to collect:

• PMU profiling samples for 120s
• With a sampling interval of 1s
• Profiles on cores 1 and 2
• TopDown metrics and render TDA sunburst chart
• PMU profiles while running the workload affinitized to cores 1 and 2.

Metrics reported by APP:

Refer to a detailed Tuning Guide Here

The APP enables PMU hardware event measurement to provide microarchitecture-level performance insight on Ampere CPUs. It is designed to answer the “why” behind performance problems by identifying microarchitectural causes such as inefficient instruction fetching and data cache misses, which are difficult to detect through application-level profiling alone.

The APP workflow is top-down and hypothesis-driven: form a hypothesis from the hot function, measure with APP profiles, then analyze using the APP HTML report with TDA sunburst charts to characterize where cycles are being spent (instruction retirement, frontend bound, backend bound). APP is most valuable when system-level bottlenecks have been characterized or ruled out and microarchitecture-level explanation is required for expert tuning.