As cloud providers push the boundaries of scalability and efficiency, innovations like nested virtualization are gaining prominence, enabling virtual machines (VMs) to host other VMs. This advanced capability provides unprecedented flexibility for cloud infrastructure. AmpereOne(R) on Arm architecture is a standout platform for this next generation of virtualization, boasting multiple innovations such as adaptive traffic management and memory quality of service (QoS) enforcement. These features enhance the performance of its mesh interconnect network and ensure fair memory bandwidth distribution. In this blog we focus on how Ampere CPUs can redefine cloud efficiency through nested virtualization.

The Importance of Nested Virtualization in Modern Cloud Deployments
Nested virtualization allows a virtual machine to act as a host, running additional VMs inside it. In simple terms, it enables running a hypervisor within a VM, creating an entire virtualized environment inside another virtualized environment. This enables workload isolation by allowing each application to run its own operating system, separate from other workloads, making microVMs essential. Workload isolation is critical for both WASM and containerized environments.

As Ampere servers continue to penetrate cloud infrastructures, the ability to support nested virtualization is critical. Cloud customers demand advanced virtualization capabilities, such as the ability to test full cloud-native environments, run multiple hypervisors, or offer custom virtualized environments, all within a single infrastructure. Ampere CPUs, with their focus on high core counts and power efficiency, are well-suited to these advanced cloud use cases.

Nested Virtualization on Arm Architecture with NV2
Nested virtualization (NV2) enables unmodified hypervisor (KVM) to run as guest hypervisors with minimum performance overhead.

Fig 1: Nested Virtualization NV2 stack.

Here’s a simplified breakdown of the nested virtualization structure:

• Host Hypervisor(L0) : The main host kernel running KVM and NV2.
• Guest Hypervisor(L1):: A VM with nested virtualization enabled, acting as a virtual host capable of running its own VMs.
• Virtual EL2 (vEL2): A pseudo state where the guest hypervisor behaves as the host hypervisor, though it operates in the EL1 state.
• Nested VM (NVM or L2): A virtual machine running inside the guest hypervisor.

Arm v8.3 NV provides ISA capabilities enabling software emulation of vEL2 in EL1. The key innovation in NV2 is the reduction of performance bottlenecks caused by VM entry/exit operations. As shown in figure2 below, NV2 on Armv8.4 architecture implements mechanisms that reduces/avoids the number of traps by:

• Instructions accessing certain special-purpose EL2 registers executed at EL1 are redefined to access the corresponding EL1 register.
• Instructions accessing certain system registers executed at EL1 are redefined to access the corresponding EL1 register.
• Loads and stores generated by transforming register accesses (for MRS or MSR instruction accessing some registers at EL1, transforms that access into a load or store respectively).

Fig 2: Mechanisms to reduce number of traps

AmpereOne’s Role in Nested Virtualization Efficiency
AmpereOne processors provide the ideal foundation for nested virtualization by leveraging two key aspects:

• High core counts: AmpereOne processors can scale up to 192 cores, making them capable of handling the added overhead that comes with running multiple layers of VMs while maintaining strong performance.
• Power efficiency: Ampere’s powerpower-efficient CPUs are well-suited for large-scale nested virtualization, where multiple virtualized layers require high compute density without increased power consumption.

For cloud providers, nested virtualization provides the following benefits:
1. The ability to test and validate OS kernel features in test environments (e.g. the automotive use case). 2. Workload isolation and security for ephemeral workloads (CI runs, WASM functions) running on a CSP's managed platform like Kubernetes (where typically the Compute nodes are VMs provisioned by the CSP).

Use Case: Automotive Industry & Nested Virtualization
One of the most compelling use cases for nested virtualization is in the automotive sector, where it plays a critical role in the development and deployment of in-vehicle applications. Nested virtualization enables OEMs to develop and test software in cloud environments and easily deploy it to vehicle hardware. This seamless cloud-to-hardware migration is essential for complex automotive systems that require functional safety and cybersecurity assurance.

Key Benefits for Automotive Applications:
1. Functional Safety: Nested virtualization provides hardware-enforced isolation between the virtualized guest operating systems (OSs), ensuring that critical safety systems remain unaffected by potential failures in non-critical systems. 2. Cybersecurity: By minimizing the Trusted Computing Base (TCB), nested virtualization reduces the attack surface, enhancing security for mission-critical systems. 3. Performance: AmpereOne’s high core count and memory efficiency, coupled with real-time, pre-emptive VM scheduling, ensure minimal boot time, memory footprint, and execution overhead—key requirements for in-vehicle systems.

For example, HARMAN’s Device Virtualization Solution has become the infrastructure of choice for automotive-grade virtualization, enabling nested virtualization to enhance security and performance in the in-vehicle domain controller. With AmpereOne processors, automotive OEMs can ensure seamless, secure, and efficient application deployment, making it a compelling solution for the future of software-defined vehicles.

Conclusion: The Future of Nested Virtualization with Ampere
The ability to run VMs inside VMs, with minimal performance trade-offs, is quickly becoming a requirement for cloud providers and industries like automotive, where complex workloads and multi-tenant environments are the norm. AmpereOne’s processors provide the performance, scalability, and efficiency needed to fully unlock the potential of nested virtualization.

With the NV2 implementation on Ampere processors, cloud and enterprise workloads can now enjoy the benefits of multi-layered virtualization without the overhead that has traditionally hampered nested environments. Whether for developing complex cloud-native applications, running multiple hypervisors, or enabling automotive innovations, Ampere CPUs are set to play a key role in advancing the future of cloud infrastructure.

As more industries transition to cloud-native architectures, nested virtualization on Ampere CPUs will enable greater flexibility, resource efficiency, and security—ultimately unlocking the infinite layers of virtualization possibilities for modern workloads.