Virtualization Internals Part 5 - KVM Internals: From VM Creation to Guest Execution

Welcome to chapter 5 of virtualization internals. This chapter provides you with an insight into hardware virtualization and particularly with KVM. The previous chapter described some technical background on how QEMU works, which is considered as an important foundation for what we will be learning today.

Introducing KVM #

Kernel Virtual Machine (KVM) is an open source hypervisor that was created to leverage modern hardware technologies available today (VT-x/AMD-V, EPT/NPT, …). KVM is built into Linux, if you’ve got Linux, you’ve got KVM along with it. Because of this, KVM benefits from all the components of the linux kernel such as the memory manager, the process scheduler, device drivers, fixes, etc. We have seen before that KVM works in conjuction with QEMU, the former deals with CPU and memory virtualization while the later handles I/O device emulation.

This chapter will focus on Intel, nevertheless, the information acquired will still holds true for AMD. I will try as much as I can to highlight the differences between the two architectures. Speaking of witch, Intel’s technology for hardware virtualization is called VT-x, while AMD’s one is called AMD-V. If you see the string VMX (Virtual Machine Extensions), that’s Intel, while SVM (Secure Virtual Machine) denotes AMD.

Before we start exploring KVM code, let’s clone the linux kernel source first. Just to avoid pulling the whole git repo history changes from the beginning of time :), let’s clone only a specific tag (v6.18 is the last LTS kernel as of writing). The KVM part from the linux kernel is located in arch/x86/kvm and virt/kvm.

git clone --depth 1 --branch v6.18 https://github.com/torvalds/linux.git

Keep in mind that we will be switching between QEMU (which you cloned previsouly) and KVM code during this post.

KVM Initialization from Kernel #

When you load the KVM kernel module, it turns the linux kernel into a hypervisor. Each virtual machine is contained in a qemu-kvm process. There is a common kernel module called kvm.ko and hardware dependent modules such as kvm-intel.ko or kvm-amd.ko.

Usually kvm.ko is loaded automatically as a dependency of the CPU-specific module, because kvm-intel.ko calls functions exported by kvm.ko, such as kvm_init() and kvm_x86_vendor_init(). During the kernel build, depmod records this dependencys, so whenever you load the Intel/AMD module, it will load the common one first.

modprobe kvm_intel
        |
        v
kvm.ko: kvm_x86_init() @ `arch/x86/kvm/x86.c`
        |
        v
kvm-intel.ko: vt_init()
        |
        +--> vmx_init() @ `arch/x86/kvm/vmx/main.c`
        |      |
        |      +--> inspect VMX capabilities
        |      +--> kvm_x86_vendor_init()
        |      +--> vmx_hardware_setup()
        |
        +--> kvm_init()
               |
               +--> enable VMX on every CPU
               +--> register /dev/kvm

kvm_x86_init() performs architecture-wide initialization that does not depend specifically on Intel or AMD, it calculates the sizes of supported extended CPU state, such as FPU/XSAVE state. Additionslly, it performs vendor-independent part of KVM’s x86 MMU initialization:

• Apply the NX huge-page mitigation when CPU is vulnerable to the Intel iTLB multihit vulnerability.
• Snapshots whether the user permits the Two Dimensional Paging (TDP) MMU, TDP is just another name for nested paging or Second Level Address Translation (SLAT), which we will take a closer look at it later.
• And the user’s requested MMIO caching setting plus the host CPU’s raw MAXPHYADDR. The raw physical-address width is important when KVM constructs shadow page-table entries and decides which SPTE bits are physical-address bits versus reserved or software-defined bits. It deliberately uses raw CPUID because SME/MKTME can make boot_cpu_data.x86_phys_bits smaller by removing encryption key-ID bits.

The Intel module starts at vt_init(), rather than vmx_init(), because this module can support both conventional VMX guests and Intel TDX guests. Its principal sequence is:

  r = vmx_init();
  r = tdx_bringup();
  r = kvm_init(sizeof(struct vcpu_vmx), ...);

The important ordering rule is that kvm_init() comes last because it eventually exposes /dev/kvm to userspace. vmx_init() does four major things:

1. Calls kvm_is_vmx_supported() to verify that the CPU and firmware permit VMX.
2. Calls hv_init_evmcs() to initialize the eVMCS structure, eVMCS is Hyper-V’s shared-memory alternative representation of VMCS12.
3. Calls setup_vmcs_config() to read the VMX capability MSRs and determine valid VMCS control bits.
4. Calls kvm_x86_vendor_init(&vt_init_ops).

Within the vt_init_ops (which contains operations needed while loading the Intel KVM module), we have the vt_init_ops object that supplies the Intel implementation of KVM’s architecture operations:

    ...
   .enable_virtualization_cpu = vmx_enable_virtualization_cpu,
   .vcpu_create              = vt_op(vcpu_create),
   .vcpu_load                = vt_op(vcpu_load),

This large operation table is the central connection between common KVM code and Intel VMX code. Like QEMU object model, this is essentially runtime polymorphism implemented manually in C using function pointers.

Going inside kvm_x86_vendor_init(), it ensures that no other vendor module is already loaded, then:

• Checks required CPU features such as FPU and FXSR.
• Allocates the x86 instruction-emulator cache. The name x86_emulator_cache() is slightly misleading. It is primarily a Linux slab allocator for emulator-context objects. It is not a cache of previously decoded guest instructions. At this stage KVM creates only the allocator, a pool capable of efficiently allocating fixed-size struct x86_emulate_ctxt objects. It does not allocate one immediately.
• Allocates per-CPU MSR state.
• Initializes the vendor-dependent MMU layer (Reset vendor-sensitive SPTE state and allocate MMU caches).
• Calls ops->hardware_setup().
• Installs the Intel operation table into KVM’s static-call dispatch.
• Checks that every online CPU is compatible.

For Intel, hardware_setup() resolves to vmx_hardware_setup(). This decides whether features such as EPT, VPID, unrestricted guests, APIC virtualization, PML, and nested VMX can be used. It also configures KVM’s MMU for EPT. Near the end, alloc_kvm_area() allocates one VMXON region for every possible CPU.

Back in vt_init(), KVM calls tdx_bringup() (we will skip all TDX related code in this article), then calls kvm_init() to initialize the common userspace-facing KVM infrastructure:

• The slab cache used to allocate struct vcpu_vmx.
• Per-CPU vCPU kick masks.
• IRQFD and asynchronous page-fault support.
• VFIO and guest-memory support.
• Preemption callbacks.
• Hardware virtualization.
• Finally, exposes /dev/kvm.

The final registration is deliberately last:

 r = misc_register(&kvm_dev);

The device definition and its file operations are in linux/virt/kvm/kvm_main.c:5557:

static struct file_operations kvm_chardev_ops = {
	.unlocked_ioctl = kvm_dev_ioctl,
	.llseek		= noop_llseek,
	KVM_COMPAT(kvm_dev_ioctl), // <-- KVM ioctl dispatcher
};

static struct miscdevice kvm_dev = {
	KVM_MINOR,
	"kvm",
	&kvm_chardev_ops,
};

Because enable_virt_at_load defaults to true, kvm_init_virtualization() calls kvm_enable_virtualization() which registers a CPU-hotplug state. The callback runs on each online CPU:

kvm_online_cpu()
-> kvm_enable_virtualization_cpu()
   -> kvm_arch_enable_virtualization_cpu()
      -> vmx_enable_virtualization_cpu()
         -> kvm_cpu_vmxon()
            -> VMXON instruction

The x86 dispatch occurs in linux/arch/x86/kvm/x86.c:13076, and the Intel implementation is in linux/arch/x86/kvm/vmx/vmx.c:2841.

vmx_enable_virtualization_cpu():

1. Obtains that CPU’s VMXON region.
2. Sets CR4.VMXE.
3. Executes VMXON using the region’s physical address.

Only after all this succeeds does KVM register /dev/kvm.

KVM Initialization from QEMU #

In the previous chapter, we have seen that KVM is registered in QEMU object model as TYPE_KVM_ACCEL and represented with the KVMState structure, it inherits from AccelState.

struct KVMState
{
    AccelState parent_obj;

    int nr_slots;
    int fd;
    int vmfd;
    int coalesced_mmio;
    ...
    /* For "info mtree -f" to tell if an MR is registered in KVM */
    int nr_as;
    struct KVMAs {
        KVMMemoryListener *ml;
        AddressSpace *as;
    } *as;
};

To start QEMU in KVM mode, append -enable-kvm to the additional start options, or you can add the accel=kvm of the -machine option. The KVM accelerator type have its instance init function points to kvm_accel_class_init which set the acceletor object ac.init_machine member to kvm_init. Here is the code flow that traces from the main function to the kvm initialization function:

main()
-> qemu_init()
   -> configure_accelerators()
      -> qemu_opts_foreach(..., do_configure_accelerator, ...)
         -> do_configure_accelerator()
            -> accel_init_machine()
               -> acc->init_machine()
                  -> kvm_init()

The do_configure_accelerator() function calls qemu_opt_get() by supplying it the qemu cmd line options (QemuOps), then the returned string is given as argument to accel_find() which returns the ac (AccelClass) object. The ACCEL macro turn the AccelClass object to an AccelState object.

    s->fd = qemu_open("/dev/kvm", O_RDWR);
    if (s->fd == -1) {
        fprintf(stderr, "Could not access KVM kernel module: %m\n");
        ret = -errno;
        goto err;
    }

Opening /dev/kvm does not create a virtual machine. It gives QEMU a system KVM file descriptor, stored in s->fd, on which it can issue system-wide ioctls. There is no KVM-specific .open callback; the important entry point is kvm_dev_ioctl() from virt/kvm/kvm_main.c.

QEMU first verifies that it understands the kernel API and queries the capabilities it needs. It then selects the machine type and asks the kernel to create a VM:

ret = kvm_ioctl(s, KVM_GET_API_VERSION, 0);
...
type = find_kvm_machine_type(ms);
ret = do_kvm_create_vm(s, type);
...
s->vmfd = ret;

do_kvm_create_vm() ultimately performs:

ret = kvm_ioctl(s, KVM_CREATE_VM, type);

For a regular x86 VM, type is normally KVM_X86_DEFAULT_VM, whose value is zero. The returned value is another file descriptor, s->vmfd, dedicated to this particular VM. We now have two interfaces with different responsibilities:

/dev/kvm fd (s->fd)
  +-- KVM_GET_API_VERSION
  +-- KVM_CHECK_EXTENSION
  `-- KVM_CREATE_VM
          |
          `--> VM fd (s->vmfd)
                 +-- KVM_SET_USER_MEMORY_REGION
                 +-- KVM_CREATE_IRQCHIP
                 `-- KVM_CREATE_VCPU

The Birth of a VM #

Because the ioctl is issued on /dev/kvm, it is dispatched through kvm_chardev_ops to kvm_dev_ioctl():

case KVM_CREATE_VM:
    r = kvm_dev_ioctl_create_vm(arg);
    break;

The kernel reserves a new file descriptor and calls kvm_create_vm(). The first interesting operation is the allocation of the VM object:

struct kvm *kvm = kvm_arch_alloc_vm();

On x86, the allocation size is not hardcoded to sizeof(struct kvm). It comes from the selected vendor operations:

static inline struct kvm *kvm_arch_alloc_vm(void)
{
    return kvzalloc(kvm_x86_ops.vm_size, GFP_KERNEL_ACCOUNT);
}

The Intel table sets .vm_size = sizeof(struct kvm_vmx), and struct kvm_vmx embeds the common object as its first member:

struct kvm_vmx {
    struct kvm kvm;

    unsigned int tss_addr;
    bool ept_identity_pagetable_done;
    ...
};

This is another object-oriented pattern implemented in C. Common code receives a struct kvm *, while VMX code can recover the enclosing struct kvm_vmx with container_of().

kvm_create_vm() then initializes the vendor-independent VM infrastructure: locks, reference counting, the future vCPU array, IRQ routing, I/O buses, SRCU domains and memory-slot metadata. It also attaches the VM to the address space of the QEMU process:

mmgrab(current->mm);
kvm->mm = current->mm;

This relationship matters because guest RAM will later be backed by mappings in QEMU’s address space. No guest RAM is registered yet; KVM merely prepares the memory-slot structures that will describe it. Each address space has two sets of slots so KVM can construct an updated set and switch the active pointer safely while readers use the old one.

Next, common KVM enters the x86-specific layer:

r = kvm_arch_init_vm(kvm, type);

kvm_arch_init_vm() validates the VM type, initializes page tracking and the per-VM MMU, and invokes the vendor callback:

ret = kvm_x86_call(vm_init)(kvm);

On Intel this dispatches through vt_vm_init(). A normal VM goes to vmx_vm_init(), while a TDX VM goes to tdx_vm_init(). For a normal VM, vmx_vm_init() configures a small amount of per-VM policy such as PAUSE-loop exiting, L1TF mitigation behavior and PML dirty-log size. It does not allocate a VMCS: a VMCS represents a virtual CPU, so it will be allocated later by KVM_CREATE_VCPU.

After the architecture-specific setup, KVM registers an MMU notifier for QEMU’s address space. This allows KVM to invalidate corresponding MMU or EPT mappings if QEMU’s host mappings are unmapped or otherwise changed. The VM is then added to KVM’s global vm_list, and per-VM debugfs and power-management state are initialized.

Finally, kvm_dev_ioctl_create_vm() wraps the struct kvm in an anonymous-inode file:

file = anon_inode_getfile("kvm-vm", &kvm_vm_fops, kvm, O_RDWR);
fd_install(fd, file);
return fd;

The returned VM fd is therefore not another device node under /dev. Its file’s private data points to the new struct kvm, and its ioctls are dispatched through kvm_vm_fops to kvm_vm_ioctl().

At the end of KVM_CREATE_VM, KVM has a kernel object representing the VM, but the machine still cannot run: it has no registered guest memory, no interrupt controller and no vCPUs. The next setup stages are KVM_SET_USER_MEMORY_REGION and KVM_CREATE_VCPU.

Exposing Guest RAM to KVM #

QEMU owns the virtual machine’s RAM. It allocates host virtual memory in its own process and builds a memory topology from MemoryRegion objects. KVM does not automatically know which QEMU mapping corresponds to a guest physical address, so QEMU must register each RAM range with the VM fd returned by KVM_CREATE_VM.

The classic KVM API describes one such range with struct kvm_userspace_memory_region, defined in include/uapi/linux/kvm.h:

struct kvm_userspace_memory_region {
    __u32 slot;
    __u32 flags;
    __u64 guest_phys_addr;
    __u64 memory_size;
    __u64 userspace_addr;
};

The fields mean:

• slot: the numerical identity of the region. The low 16 bits are the slot ID and, when multiple guest address spaces are supported, the high 16 bits contain the address-space ID.
• flags: properties such as KVM_MEM_READONLY or KVM_MEM_LOG_DIRTY_PAGES.
• guest_phys_addr: where the range begins in the guest physical address space.
• memory_size: the size of the range in bytes. Setting it to zero removes an existing slot.
• userspace_addr: the starting host virtual address inside QEMU where the RAM is mapped.

For example, the following describes 512 MiB of guest RAM starting at GPA zero and backed by a QEMU mapping at 0x7f0000000000:

struct kvm_userspace_memory_region region = {
    .slot = 0,
    .flags = 0,
    .guest_phys_addr = 0,
    .memory_size = 512 * 1024 * 1024,
    .userspace_addr = 0x7f0000000000,
};

ioctl(vmfd, KVM_SET_USER_MEMORY_REGION, &region);

This does not copy 512 MiB into the kernel. It records a relationship:

guest physical range             QEMU host virtual range
0x00000000 - 0x1fffffff   <-->   0x7f0000000000 - 0x7f001fffffff

From QEMU’s memory topology to a KVM slot #

QEMU does not normally issue this ioctl directly from board code. After creating the VM, kvm_init() registers a KVMMemoryListener on address_space_memory:

kvm_memory_listener_register(s, &s->memory_listener,
                             &address_space_memory, 0, "kvm-memory");

The listener connects changes in QEMU’s memory topology to three callbacks in accel/kvm/kvm-all.c:

kml->listener.region_add = kvm_region_add;
kml->listener.region_del = kvm_region_del;
kml->listener.commit = kvm_region_commit;

When machine initialization adds RAM to QEMU’s system address space, kvm_region_add() queues the corresponding MemoryRegionSection. At the end of the memory transaction, kvm_region_commit() processes deletions first and additions second, then calls kvm_set_phys_mem() for every changed section. Batching changes this way prevents QEMU from exposing a partially updated memory topology to KVM.

kvm_set_phys_mem() performs the conversion from QEMU’s representation to KVM’s representation. For an added RAM section, it:

1. Aligns the guest address and size to host-page boundaries.
2. Calculates the offset of the section inside its MemoryRegion.
3. Obtains the QEMU host pointer with memory_region_get_ram_ptr().
4. Splits very large ranges if they exceed KVM’s maximum slot size.
5. Allocates a QEMU-side KVMSlot and assigns a free slot number.
6. Calls kvm_set_user_memory_region().

The relevant setup is:

mem->memory_size = slot_size;
mem->start_addr = start_addr;
mem->ram = ram;
mem->flags = kvm_mem_flags(mr);

err = kvm_set_user_memory_region(kml, mem, true);

KVMSlot is QEMU’s cached description of a slot; it is not the kernel’s struct kvm_memory_slot. QEMU translates it into the UAPI structure as follows:

mem.slot = slot->slot | (kml->as_id << 16);
mem.guest_phys_addr = slot->start_addr;
mem.memory_size = slot->memory_size;
mem.userspace_addr = (unsigned long)slot->ram;
mem.flags = slot->flags;

ret = kvm_vm_ioctl(s, KVM_SET_USER_MEMORY_REGION, &mem);

Modern QEMU uses KVM_SET_USER_MEMORY_REGION2 when guest-memfd support is available. The second version adds guest_memfd and guest_memfd_offset for private/protected memory, but the five fields above and the normal shared-RAM mechanism remain the same.

Removing RAM follows the same path, except QEMU finds the existing slot, synchronizes dirty-page information if necessary, sets memory_size to zero, and issues the ioctl again. QEMU can also update flags, for example to enable dirty logging during live migration.

The ioctl path inside KVM #

Unlike KVM_CREATE_VM, this ioctl is sent to the per-VM fd, so it enters kvm_vm_ioctl() in virt/kvm/kvm_main.c:

kvm_vm_ioctl() copies the userspace structure into the kernel. Internally, the old UAPI structure is expanded into struct kvm_userspace_memory_region2, with the extra fields cleared. kvm_vm_ioctl_set_memory_region() then takes kvm->slots_lock, serializing memory-slot changes for this VM.

kvm_set_memory_region() validates that the size, guest address and QEMU address are page aligned, checks the userspace range with access_ok(), rejects integer overflow and overlapping guest ranges, and separates slot into an address-space ID and slot ID:

as_id = mem->slot >> 16;
id = (u16)mem->slot;

base_gfn = mem->guest_phys_addr >> PAGE_SHIFT;
npages = mem->memory_size >> PAGE_SHIFT;

It then classifies the operation as one of four changes:

• KVM_MR_CREATE: a non-empty range for a previously unused slot.
• KVM_MR_DELETE: memory_size is zero for an existing slot.
• KVM_MR_MOVE: the same slot is moved to another guest physical range.
• KVM_MR_FLAGS_ONLY: only properties such as dirty logging change.

For a new range, KVM allocates its internal slot and stores the essential mapping information:

new->base_gfn = base_gfn;
new->npages = npages;
new->flags = mem->flags;
new->userspace_addr = mem->userspace_addr;

On x86, kvm_arch_prepare_memory_region() verifies that the GFN range is supported and allocates per-slot MMU metadata, including reverse-map data when needed, huge-page information and page-tracking metadata.

Why KVM keeps two memslot sets #

vCPU threads can look up memory while QEMU changes the memory topology. Modifying the active slot tree in place would let a vCPU observe a half-written structure. Instead, KVM updates the inactive struct kvm_memslots, publishes it with rcu_assign_pointer(), waits for old SRCU readers, and then updates the now-inactive copy.

For creation, the simplified operation is:

Deletion and movement require an extra safety step. KVM first publishes an INVALID version of the old slot, preventing new mappings from being created, and calls kvm_arch_flush_shadow_memslot() to remove existing shadow-page-table or EPT mappings. Only then can it remove or move the slot without a vCPU retaining a translation to memory that no longer belongs there.

A memslot is not an EPT mapping #

Registering a slot does not normally pin every QEMU page and does not eagerly populate EPT. A memslot is metadata that lets KVM answer: “Which QEMU address backs this guest frame number?”

For a GFN inside a slot, KVM calculates the HVA using:

offset = gfn - slot->base_gfn;
hva = slot->userspace_addr + offset * PAGE_SIZE;

Including the offset within the page, the relationship is simply:

HVA = userspace_addr + (GPA - guest_phys_addr)

The eventual translation is:

The last steps happen lazily. When a vCPU first accesses a GPA with no EPT translation, the CPU generates an EPT violation. KVM finds the memslot, calls __kvm_faultin_pfn() to resolve or fault in the QEMU page, and then kvm_tdp_mmu_map() installs the EPT entry. This demand-driven design avoids pinning and mapping all guest RAM up front.

After all RAM regions are registered, the VM has a guest physical memory layout, but it still has no virtual processor. The next operation is KVM_CREATE_VCPU, which allocates struct kvm_vcpu, the Intel struct vcpu_vmx, its emulator context and its VMCS.

Giving the VM a Brain #

A KVM VM and a KVM vCPU are separate objects. KVM_CREATE_VM created the container that owns memory slots, interrupt routing and the list of virtual processors. KVM_CREATE_VCPU now creates one processor inside that container. The ioctl returns a new file descriptor that QEMU will use to configure and run that processor.

The descriptor hierarchy now has three levels:

/dev/kvm fd (KVMState.fd)
  |
  `-- VM fd (KVMState.vmfd)
        |
        +-- vCPU 0 fd (CPUState.kvm_fd)
        +-- vCPU 1 fd (CPUState.kvm_fd)
        `-- ...

System-wide ioctls use the /dev/kvm fd, VM-wide ioctls use the VM fd, and processor-specific ioctls use a vCPU fd.

Creating the vCPU from QEMU #

QEMU normally gives every virtual CPU its own host thread. The KVM vCPU thread starts in accel/kvm/kvm-accel-ops.c and calls kvm_init_vcpu() before entering its execution loop:

r = kvm_init_vcpu(cpu, &error_fatal);
kvm_init_cpu_signals(cpu);
...
r = kvm_cpu_exec(cpu);

kvm_init_vcpu() calls kvm_create_vcpu() in accel/kvm/kvm-all.c. QEMU first calculates an architecture-specific vCPU ID and then issues the ioctl on the VM fd:

unsigned long vcpu_id = kvm_arch_vcpu_id(cpu);

kvm_fd = kvm_vm_ioctl(s, KVM_CREATE_VCPU, vcpu_id);
cpu->kvm_fd = kvm_fd;

The argument is an identifier chosen by userspace. On x86 it corresponds to the architecture’s CPU identity and is commonly related to the APIC ID. It should not be confused with QEMU’s cpu_index, nor with KVM’s later, densely allocated vcpu_idx.

QEMU can also retain, or “park,” a vCPU fd when a virtual CPU is unplugged. If the same vCPU is added again, QEMU reuses the parked fd rather than trying to create a second kernel object with the same ID.

Entering the kernel #

Because KVM_CREATE_VCPU is issued on the VM fd, it is dispatched by kvm_vm_ioctl() in virt/kvm/kvm_main.c:

case KVM_CREATE_VCPU:
    r = kvm_vm_ioctl_create_vcpu(kvm, arg);
    break;

The overall kernel path is:

KVM_CREATE_VCPU
      |
      v
kvm_vm_ioctl_create_vcpu()
      |
      +--> reserve and allocate the common vCPU object
      +--> kvm_vcpu_init()
      +--> kvm_arch_vcpu_create()
      |       |
      |       `--> vmx_vcpu_create()
      +--> publish the vCPU in its VM
      +--> create_vcpu_fd()
      `--> kvm_arch_vcpu_postcreate()

kvm_vm_ioctl_create_vcpu() first validates the requested ID and checks that the VM has not reached its maximum number of vCPUs. It calls kvm_arch_vcpu_precreate() so the x86 and VMX layers can reject an unsupported configuration. It then increments kvm->created_vcpus to reserve a vCPU while allocation takes place.

KVM keeps two different identifiers:

• vcpu_id is the ID supplied by QEMU. It can be sparse and has architectural meaning.
• vcpu_idx is a dense, zero-based index assigned when KVM publishes the vCPU in kvm->vcpu_array.

The common code allocates the vCPU from the slab cache created by kvm_init():

vcpu = kmem_cache_zalloc(kvm_vcpu_cache,
                         GFP_KERNEL_ACCOUNT);

The pointer has type struct kvm_vcpu *, but the cache’s object size was supplied by the Intel module:

kvm_init(sizeof(struct vcpu_vmx),
         __alignof__(struct vcpu_vmx), THIS_MODULE);

This works because Intel embeds the common vCPU object at offset zero:

struct vcpu_vmx {
    struct kvm_vcpu vcpu;
    struct vcpu_vt vt;
    ...
    struct loaded_vmcs vmcs01;
    struct loaded_vmcs *loaded_vmcs;
};

The layout follows the same C polymorphism pattern used for the VM object:

Common code operates on struct kvm_vcpu *. Intel code converts it to struct vcpu_vmx * when it needs VMX-specific fields.

Before entering architecture-specific initialization, KVM allocates a zeroed page for struct kvm_run:

BUILD_BUG_ON(sizeof(struct kvm_run) > PAGE_SIZE);
page = alloc_page(GFP_KERNEL_ACCOUNT | __GFP_ZERO);
vcpu->run = page_address(page);

This page will become a shared communication area between QEMU and KVM. QEMU can write run controls into it before KVM_RUN, and KVM can write the reason and details of a userspace-visible VM exit into it. The page is allocated now but mapped into QEMU only after KVM_CREATE_VCPU returns.

Common vCPU initialization #

kvm_vcpu_init() initializes state that is independent of Intel or AMD:

vcpu->cpu = -1;
vcpu->kvm = kvm;
vcpu->vcpu_id = id;
vcpu->ready = false;
vcpu->last_used_slot = NULL;

It also initializes the vCPU mutex, wait state, asynchronous page-fault state, preemption notifier, scheduling information and statistics name. A value of -1 in vcpu->cpu means that this virtual processor is not currently loaded on a physical CPU.

x86-specific initialization #

The common code next calls:

r = kvm_arch_vcpu_create(vcpu);

The x86 implementation in arch/x86/kvm/x86.c creates the software state needed by an x86 processor. This includes:

• Per-vCPU MMU state.
• A local APIC.
• A page used for port-I/O data.
• Machine-check and PMU state.
• Asynchronous page-fault tracking.
• Guest FPU and XSAVE state.
• The instruction-emulator context.

The emulator slab allocator discussed earlier is finally used here:

if (!alloc_emulate_ctxt(vcpu))
    goto free_wbinvd_dirty_mask;

Each vCPU receives one struct x86_emulate_ctxt, which is reused whenever KVM must emulate a guest instruction.

After the common x86 allocations, KVM invokes the vendor callback:

r = kvm_x86_call(vcpu_create)(vcpu);

For Intel, dispatch proceeds through vt_vcpu_create(). A normal VM calls vmx_vcpu_create(), while a TDX VM calls tdx_vcpu_create().

Allocating the Intel VMCS #

vmx_vcpu_create() in arch/x86/kvm/vmx/vmx.c allocates Intel-specific resources. Among other things, it allocates a VPID, an optional PML page and the primary VMCS:

vmx->vpid = allocate_vpid();
...
err = alloc_loaded_vmcs(&vmx->vmcs01);
...
vmx->loaded_vmcs = &vmx->vmcs01;

A VPID, or Virtual Processor Identifier, tags guest TLB entries. It allows the CPU to distinguish translations belonging to different virtual processors and reduces the need for broad TLB flushes.

The PML, or Page Modification Logging is a hardware-assisted log of guest-physical pages that the vCPU has written to. Its main purpose is to make KVM’s dirty-page tracking more efficient (for example during live migration). Without PML, KVM may need to write-protect guest pages and take a VM exit on the first write to each page.

vmcs01 is the VMCS used by the host KVM hypervisor, L0, to run the normal guest, L1:

Linux/KVM host (L0)
        |
        `-- ordinary guest (L1) uses vmcs01

If that guest later runs a nested guest, KVM creates additional nested state, including vmcs02, for running L2.

alloc_loaded_vmcs() performs the initial VMCS allocation:

1. It allocates a page-sized, properly aligned VMCS region.
2. It writes the VMCS revision ID reported by IA32_VMX_BASIC.
3. It executes VMCLEAR to place the VMCS in a valid, clear state.
4. It records that the VMCS is not loaded on any CPU and has not been launched.
5. It allocates an MSR bitmap when the CPU supports one.
6. It initializes KVM’s cached host state and VMCS control values.

The MSR bitmap tells the processor which guest RDMSR and WRMSR operations should trigger VM exits. KVM initially fills the bitmap with ones, meaning that accesses are intercepted, and later clears selected bits when direct guest access is safe.

At this stage the VMCS exists, but the guest has not executed. The launched field remains zero until a successful VMLAUNCH occurs.

Establishing reset state #

After VMX allocation succeeds, the x86 layer temporarily loads the vCPU and initializes its architectural reset state:

vcpu_load(vcpu);
kvm_vcpu_after_set_cpuid(vcpu);
kvm_set_tsc_khz(vcpu, vcpu->kvm->arch.default_tsc_khz);
kvm_vcpu_reset(vcpu, false);
kvm_init_mmu(vcpu);
vcpu_put(vcpu);

This creates a valid initial kernel representation of the processor. QEMU will subsequently send the selected CPU model, CPUID leaves, registers, special registers and MSRs through vCPU ioctls before running it.

Publishing the vCPU and returning its fd #

Once initialization succeeds, KVM assigns the dense index and inserts the vCPU into the VM’s xarray:

vcpu->vcpu_idx = atomic_read(&kvm->online_vcpus);
xa_insert(&kvm->vcpu_array, vcpu->vcpu_idx, vcpu,
          GFP_KERNEL_ACCOUNT);

KVM publishes the pointer before incrementing online_vcpus, with a memory barrier between the two operations. This prevents another kernel thread from observing a vCPU count for which the corresponding pointer is not yet visible.

KVM then creates an anonymous-inode fd:

return anon_inode_getfd(name, &kvm_vcpu_fops, vcpu,
                        O_RDWR | O_CLOEXEC);

The vCPU fd’s private data points to struct kvm_vcpu, and its file operations provide the two interfaces QEMU needs next:

static struct file_operations kvm_vcpu_fops = {
    .release        = kvm_vcpu_release,
    .unlocked_ioctl = kvm_vcpu_ioctl,
    .mmap           = kvm_vcpu_mmap,
};

Processor-specific operations such as KVM_SET_CPUID2, KVM_SET_REGS, KVM_SET_SREGS and KVM_RUN are all issued on this fd.

Mapping struct kvm_run into QEMU #

After receiving the vCPU fd, QEMU asks /dev/kvm how large the shared vCPU mapping must be:

mmap_size = kvm_ioctl(s, KVM_GET_VCPU_MMAP_SIZE, 0);

QEMU then maps the vCPU fd:

cpu->kvm_run = mmap(NULL, mmap_size,
                    PROT_READ | PROT_WRITE, MAP_SHARED,
                    cpu->kvm_fd, 0);

On x86, the mapping normally covers three pages:

page offset 0: struct kvm_run
page offset 1: port-I/O data
page offset 2: coalesced-MMIO ring

The mapping is backed by kvm_vcpu_mmap(). When QEMU faults a mapped page for the first time, kvm_vcpu_fault() selects the corresponding kernel page from the mapping offset.

struct kvm_run, defined in include/uapi/linux/kvm.h, is bidirectional. Before KVM_RUN, QEMU can set fields such as immediate_exit and request_interrupt_window. When KVM must return to userspace, it fills fields such as exit_reason and the matching union member:

KVM_EXIT_IO       port, direction, size and data offset
KVM_EXIT_MMIO     guest physical address, data and direction
KVM_EXIT_DEBUG    debug-exit information
KVM_EXIT_FAIL_ENTRY
KVM_EXIT_INTERNAL_ERROR

Sharing this page avoids copying a large result structure between QEMU and KVM on every userspace exit.

Final QEMU-side CPU configuration #

QEMU completes initialization with kvm_arch_init_vcpu(). On x86, this function constructs the CPUID table for the selected virtual CPU model and sends it through the vCPU fd:

r = kvm_vcpu_ioctl(cs, KVM_SET_CPUID2, &cpuid_data);

It also configures details such as TSC frequency and optional Hyper-V features. QEMU later synchronizes general registers, special registers and MSRs with ioctls such as KVM_SET_REGS, KVM_SET_SREGS and KVM_SET_MSRS.

At the end of KVM_CREATE_VCPU and its QEMU-side initialization, the state is:

VM object             created
guest memory slots    registered
vCPU object           created
emulator context      allocated
guest FPU state       allocated
VMCS                  allocated and cleared
struct kvm_run        mapped into QEMU
guest execution       not started

What’s left now is to run the vCPU with KVM_RUN. That path loads the vCPU on a host CPU, prepares the VMCS, enters the guest with VMLAUNCH or VMRESUME, handles VM exits and decides whether an exit can be handled inside the kernel or must be returned to QEMU.

The vCPU Comes Alive with KVM_RUN #

KVM_RUN is where virtualization finally becomes active. All previous steps created and configured objects: the VM, its memory slots, the vCPU, and its VMCS. When QEMU issues KVM_RUN on a vCPU fd, the calling host thread enters the kernel and normally does not return until KVM needs help from userspace, a signal interrupts the thread, or an unrecoverable error occurs.

One KVM_RUN ioctl can contain many guest entries and VM exits. A VM exit does not automatically mean a return to QEMU. KVM first tries to handle the exit in the kernel and resumes the guest when possible.

QEMU’s vCPU execution loop #

The QEMU vCPU thread repeatedly calls kvm_cpu_exec() from accel/kvm/kvm-all.c. Before the ioctl, QEMU synchronizes any CPU state that has changed in userspace:

if (cpu->vcpu_dirty) {
    ret = kvm_arch_put_registers(cpu, KVM_PUT_RUNTIME_STATE, &err);
    ...
    cpu->vcpu_dirty = false;
}

kvm_arch_pre_run(cpu, run);
run_ret = kvm_vcpu_ioctl(cpu, KVM_RUN, 0);

On x86, register synchronization uses ioctls such as KVM_SET_REGS, KVM_SET_SREGS and KVM_SET_MSRS. kvm_arch_pre_run() prepares architecture-specific state, including interrupt injection when necessary.

QEMU releases the Big QEMU Lock before entering this loop. The vCPU can therefore execute in parallel with other vCPUs and many device-model operations. If another QEMU thread needs the vCPU to leave KVM, it sets an exit request and kicks the vCPU thread. Depending on the available KVM capability, the kick is observed through the shared immediate_exit field or through a signal that interrupts the ioctl.

Entering the vCPU ioctl dispatcher #

KVM_RUN is sent to the vCPU fd, whose .unlocked_ioctl callback is kvm_vcpu_ioctl() in virt/kvm/kvm_main.c. KVM first verifies that the caller belongs to the VM’s QEMU address space, waits for vCPU creation to be completely visible, and locks vcpu->mutex so that two host threads cannot run or modify the same vCPU concurrently.

The KVM_RUN case records which host task is running the vCPU and then enters the architecture-specific code:

vcpu->wants_to_run =
    !READ_ONCE(vcpu->run->immediate_exit__unsafe);

r = kvm_arch_vcpu_ioctl_run(vcpu);

vcpu->wants_to_run = false;

The host task identity is useful for directed yielding and diagnostics. If QEMU moves a vCPU fd to another thread, KVM notices the PID change and updates architecture-specific state before continuing.

The complete path is approximately:

Preparing x86 state for execution #

kvm_arch_vcpu_ioctl_run() in arch/x86/kvm/x86.c prepares the vCPU before entering the internal run loop. Its main responsibilities are:

1. Finish delayed per-VM MMU initialization.
2. Load the vCPU on the current physical CPU.
3. Activate QEMU’s requested signal mask.
4. Swap the host thread’s FPU state for the guest FPU state.
5. Enter an SRCU read-side section that protects memory slots and related VM data.
6. Import register fields that QEMU marked dirty in struct kvm_run.
7. Complete any PIO or MMIO operation that previously returned to userspace.
8. Enter vcpu_run().

The FPU switch is important. The hardware FPU registers cannot simultaneously contain both QEMU’s userspace state and the guest’s state. KVM calls kvm_load_guest_fpu() before running the guest and restores the host thread’s FPU state with kvm_put_guest_fpu() before returning to QEMU.

vcpu_load() associates the virtual CPU with the physical CPU on which the QEMU thread is currently scheduled. For Intel, vmx_vcpu_load_vmcs() executes VMPTRLD when the vCPU’s VMCS is not already current on that physical CPU. If the vCPU migrated from another physical CPU, KVM clears the old association, requests a TLB flush, and refreshes per-CPU VMCS host fields such as the TSS and GDT addresses.

This association is why a vCPU is not permanently tied to one physical CPU. Linux can schedule its QEMU thread elsewhere, and KVM reloads the VMCS and host-specific state on the new CPU.

The internal KVM run loop #

vcpu_run() is a kernel loop around vcpu_enter_guest():

for (;;) {
    if (kvm_vcpu_running(vcpu))
        r = vcpu_enter_guest(vcpu);
    else
        r = vcpu_block(vcpu);

    if (r <= 0)
        break;

    ...
}

The return-value convention is central to understanding this path:

• A positive value means that KVM handled the condition and can continue its kernel run loop.
• Zero means that struct kvm_run contains an exit requiring userspace assistance.
• A negative value reports an interruption or error, such as -EINTR.

If the guest executed HLT and no event can wake it, KVM can block the vCPU thread inside the kernel instead of repeatedly returning to QEMU. A virtual interrupt, timer, signal or another wake event makes it runnable again.

Work performed before VM entry #

vcpu_enter_guest() handles work that must be completed before the processor can safely enter VMX non-root operation. KVM first services pending KVM_REQ_* requests. These requests can ask the vCPU to:

• Reload or synchronize MMU roots.
• Flush guest TLB entries.
• Update the guest clock.
• Recalculate instruction intercepts.
• Process INIT, SIPI, SMI or NMI events.
• Inject pending interrupts or exceptions.
• Update APIC virtualization state.
• Process asynchronous page-fault completion.
• Stop because the VM is being destroyed or its dirty ring is full.

After processing requests, KVM evaluates pending events and prepares interrupt or exception injection. It then calls kvm_mmu_reload() to ensure that the vCPU has valid MMU roots. With EPT enabled, this includes ensuring that the appropriate EPT root is available and that its address can be loaded into the VMCS.

Immediately before entry, KVM disables preemption and local interrupts, calls the vendor’s prepare_switch_to_guest() operation, and changes vcpu->mode to IN_GUEST_MODE. The ordering around this state change is carefully synchronized with vCPU kicks and remote TLB flushes. If a request arrives during the transition, KVM detects it and cancels the entry rather than allowing the guest to run with stale state.

The vendor dispatch is then:

exit_fastpath = kvm_x86_call(vcpu_run)(vcpu, run_flags);

For an ordinary Intel VM, this resolves through vt_vcpu_run() to vmx_vcpu_run().

Preparing the VMCS #

vmx_vcpu_run() performs the final Intel-specific preparation. It does not rewrite every VMCS field on every entry. KVM maintains register-valid and register-dirty bitmaps and only writes fields whose cached values changed. For example:

if (kvm_register_is_dirty(vcpu, VCPU_REGS_RSP))
    vmcs_writel(GUEST_RSP, vcpu->arch.regs[VCPU_REGS_RSP]);

if (kvm_register_is_dirty(vcpu, VCPU_REGS_RIP))
    vmcs_writel(GUEST_RIP, vcpu->arch.regs[VCPU_REGS_RIP]);

KVM also refreshes host state that may have changed since the previous entry, such as HOST_CR3, HOST_CR4 and debug-control state. It loads guest XSAVE state, prepares performance-monitoring state, updates the VMX preemption timer, and applies security mitigations required when crossing the host/guest boundary.

The VMCS contains two broad categories of state:

Guest-state fields
  RIP, RSP, RFLAGS
  CR0, CR3, CR4
  segment state
  interruptibility state
  guest MSRs

Host-state fields
  host CR3 and CR4
  host segment and descriptor-table state
  host RSP
  host RIP = vmx_vmexit

The VM-execution controls specify which guest operations cause VM exits. The entry and exit controls specify how the CPU transfers state across the boundary. The EPT pointer tells the processor which second-level page tables translate guest physical addresses.

VMLAUNCH versus VMRESUME #

KVM tracks whether the current VMCS has successfully entered a guest:

if (vmx->loaded_vmcs->launched)
    flags |= VMX_RUN_VMRESUME;

The first entry after VMCLEAR must use VMLAUNCH. Later entries use VMRESUME:

VMCS clear state
      |
      `--> VMLAUNCH
              |
              v
        VMCS launched state
              |
              `--> VMRESUME
                    VMRESUME
                    VMRESUME ...

Executing VMLAUNCH on an already launched VMCS, or VMRESUME on a clear VMCS, causes a VM-instruction failure. If a VMCS moves to another physical CPU and must be cleared, KVM resets its software launched flag so that the next entry uses VMLAUNCH again.

Crossing the hardware boundary #

The actual transition is implemented by __vmx_vcpu_run() in arch/x86/kvm/vmx/vmenter.S. Assembly is required because KVM must explicitly move the guest’s general-purpose registers into the physical registers immediately before entry:

mov VCPU_RCX(%reg), %rcx
mov VCPU_RDX(%reg), %rdx
...

jnc .Lvmlaunch
vmresume
...
.Lvmlaunch:
vmlaunch

The processor obtains fields such as guest RIP, RSP, control registers and segment state from the VMCS. General-purpose registers are not automatically loaded from or saved into the VMCS, so the assembly code handles them separately.

On successful VM entry, the physical CPU begins executing guest instructions in VMX non-root operation. The QEMU vCPU thread has not disappeared: its kernel execution context is suspended at the VM-entry boundary while the physical CPU runs guest state.

What hardware does on a VM exit #

When an enabled exit condition occurs, the processor performs a VM exit. Examples include an intercepted I/O instruction, CPUID, selected MSR accesses, an EPT violation, HLT, or an external interrupt.

Hardware then performs the essential state transition:

1. It records the exit reason, qualification and relevant guest state in the VMCS.
2. It saves VMCS-managed guest state.
3. It loads the host state from the VMCS.
4. It loads the VMCS host RIP and host RSP.
5. Execution continues at KVM’s vmx_vmexit assembly label.

At vmx_vmexit, KVM saves the guest’s general-purpose registers back into vcpu->arch.regs. The assembly code also restores host speculation-control state and clears registers that could otherwise retain guest values during speculative execution.

Control returns to vmx_vcpu_run(), which reads VM_EXIT_REASON, records that the VMCS has now been launched, completes interrupt bookkeeping and selects a fast-path result when possible.

Fast-path and normal exit handling #

Some frequent exits can be processed without restoring the full host execution environment. For example, KVM can consume a VMX preemption-timer exit or certain interrupt conditions and immediately re-enter the guest. This is represented by EXIT_FASTPATH_REENTER_GUEST.

If the fast path cannot completely handle the exit, vcpu_enter_guest() restores host interrupt and preemption state and calls:

r = kvm_x86_call(handle_exit)(vcpu, exit_fastpath);

For Intel this reaches vmx_handle_exit(), which dispatches on the hardware exit reason through kvm_vmx_exit_handlers[]:

[EXIT_REASON_IO_INSTRUCTION] = handle_io,
[EXIT_REASON_CPUID]          = kvm_emulate_cpuid,
[EXIT_REASON_MSR_READ]       = kvm_emulate_rdmsr,
[EXIT_REASON_MSR_WRITE]      = kvm_emulate_wrmsr,
[EXIT_REASON_HLT]            = kvm_emulate_halt,
[EXIT_REASON_EPT_VIOLATION]  = handle_ept_violation,

The handler’s return value decides where execution goes next:

Examples of exits handled inside KVM #

Many exits never become visible to QEMU:

• CPUID is usually emulated from the vCPU’s KVM CPUID table.
• Common MSR reads and writes are emulated by KVM.
• Control-register accesses update KVM’s virtual CPU and MMU state.
• An external interrupt is serviced by the host kernel, after which the guest can resume.
• An EPT violation for ordinary guest RAM resolves the memslot’s HVA to a host PFN and installs an EPT entry.
• A guest HLT can block the vCPU thread in KVM when the in-kernel LAPIC is responsible for wakeup events.

These operations return a positive value to the run loop. QEMU sees none of the intermediate VM exits.

An EPT violation is a particularly important example:

If the GPA belongs to normal RAM, the entire sequence can remain inside one KVM_RUN call.

Exits that return to QEMU #

KVM returns to userspace when the kernel cannot or should not implement the required policy. It places the reason and associated data in the shared struct kvm_run and returns zero from the ioctl.

Typical examples include:

• Port I/O that belongs to a QEMU-emulated device.
• MMIO to a QEMU-emulated device.
• A userspace-visible debug event.
• A shutdown or system event.
• An entry failure or internal error.
• An unhandled or filtered MSR access configured to exit to userspace.

For a port-I/O exit, KVM fills fields such as:

run->exit_reason = KVM_EXIT_IO;
run->io.port = port;
run->io.direction = direction;
run->io.size = size;
run->io.count = count;

For MMIO, KVM returns the guest physical address, access size, direction and up to eight bytes of data through run->mmio.

QEMU handles a userspace exit #

When the ioctl returns, kvm_cpu_exec() examines run->exit_reason:

switch (run->exit_reason) {
case KVM_EXIT_IO:
    kvm_handle_io(...);
    ret = 0;
    break;

case KVM_EXIT_MMIO:
    address_space_rw(...);
    ret = 0;
    break;

default:
    ret = kvm_arch_handle_exit(cpu, run);
    break;
}

QEMU routes PIO and MMIO through its device model. A virtual UART access may reach QEMU’s serial-device implementation; a disk-controller access may reach the emulated storage device. Architecture-specific exits such as KVM_EXIT_HLT, debug exits and selected MSR exits are handled by kvm_arch_handle_exit() in target/i386/kvm/kvm.c.

If QEMU handles the exit successfully and sets ret to zero, its do ... while loop issues KVM_RUN again immediately.

Completing an interrupted instruction #

For a userspace-assisted input or MMIO read, the guest instruction is not complete when KVM first returns. QEMU writes the emulated result into struct kvm_run and calls KVM_RUN again. At the beginning of the new call, KVM notices vcpu->arch.complete_userspace_io and invokes the saved completion callback:

This explains QEMU’s comment that it must re-enter KVM after an I/O exit even when an external exit request is pending. KVM may need the next ioctl invocation to finish the instruction before it can safely return for another reason.

Returning from KVM_RUN #

Before returning to QEMU, kvm_arch_vcpu_ioctl_run():

• Restores the QEMU thread’s FPU state.
• Copies requested synchronized registers into struct kvm_run.
• Saves post-run APIC and interrupt state.
• Leaves the memory-slot SRCU read section.
• Restores the host signal mask.
• Calls vcpu_put() to release the vCPU’s physical-CPU association state.

The common ioctl layer then clears vcpu->wants_to_run, records the userspace exit tracepoint and releases vcpu->mutex.

The complete steady-state cycle is therefore:

At this point, we have followed the complete control path from QEMU into hardware-assisted guest execution and back. In the next part, we will focus on KVM’s TDP MMU: how it manages second-level address translation, resolves EPT violations, and builds mappings from guest physical addresses to host pages.