CMPE283 : Virtualization

Assignment 1: Discovering VMX Features

• Read the VMX configuration MSRs to ascertain support capabilities/features Entry / Exit / Procbased / Secondary Procbased / Pinbased controls

• For each group of controls above, interpret and output the values read from the MSR to the system via printk(..), including if the value can be set or cleared.

Answer 1

Answer 2

Steps to create a Linux kernel module that will query various MSRs to determine virtualization features available in your CPU

System configurations
Host OS : Windows 10 x64 based processor
RAM : 8GB
Processor : Intel(R) Core(TM) i5

1. Install VMware Workstation 16 Player on host OS. (Install VM fusion if you are using Mac as Host OS)
   
2. Install Ubuntu OS as Guest OS (Ubuntu 64-bit 20.04.3)
   
3. After installation enable hardware assisted virtaulization as follows
   Go to VM Machine settings -> Processors -> Virtualization engine -> Enable the 'Virtualize Intel VT-x/EPT or AMD-V/RVI' checkbox.
   
4. Install git, build-essential, libssl-dev, flex, bison.
   
5. Clone the repository https://github.com/archana-kamath/linux.git
   
6. Update config file to get the configurations required the latested cloned version of linux.
   
7. Install all the modules and kernel code.(This step will take time depending on the your system configuration)
   
8. Reboot the VM check if the latest recently installed linux version is selected.
   
9. Create a folder cmpe283 inside linux folder and copy Makefile and cmpe283-1.c provided.
   
10. Update the cmpe283-1.c file to get all the VMX functionalities and build it. Add Modules license code too.
    
11. Run the command make and see if build is successful.
    
12. Run command 'sudo insmod cmpe283-1.ko' to load the file to kernel.
    
13. Run command 'dmesg' to get the output. Observe the if the values can be set or cleared for the 5 controls.
    
14. Run command 'sudo rmmod cmpe283-1' to load the file out of the kernel.
    
15. Commit and push the Makefile and cmpe283-1.c to git repository, after giving username and token.
    

Screen prints of the output

Pinbased and Primary Proc Based controls

Secondary Proc Based controls

Entry and Exit Based controls

Assignment 2:

Answer 1:

Archana Miyar Kamath (015276378)
Setting up Virtual machine
Code changes in cpuid.c and vmx.c.
Building the code changes.
Debugging the build issues.

Mounica Kamireddy (015949414)
Understanding the usage of int, atomic variables and u32, u64 etc
Installing nested machine and implemented test cases.
Debugging issues while installing nested virtual machine.

Answer 2

Part 1:

For CPUID leaf node %eax=0x4FFFFFFF: Return the total number of exits (all types) in %eax

Part 2:

For CPUID leaf node %eax=0x4FFFFFFE: Return the high 32 bits of the total time spent processing all exits in %ebx Return the low 32 bits of the total time spent processing all exits in %ecx %ebx and %ecx return values are measured in processor cycles, across all VCPUs

Configuration done in Assignment 1 is required to proceed with assignment 2.

Do the code changes in cpuid.c and vmx.c path : /linux/arch/x86/kvm/cpuid.c /linux/arch/x86/kvm/vmx/vmx.c

Run the below commands in order

sudo make modules sudo make INSTALL_MOD_STRIP=1 modules_install sudo make install

lsmod | grep kvm // These below two files should not be there, if so remove them sudo rmmod kvm_intel sudo rmmod kvm

lsmod | grep kvm (kvm_intel and kvm should not be in response)

sudo modprobe kvm sudo modprobe kvm_intel

Reboot the VM

Steps to test

Set up an VM inside the host machine to test the changes. Install the following - virtinst, libvirt-clients, virt-top, qemu-kvm, libvirt-daemon, libvirt-daemon-systems, virt-manager, bridge-utils

Run the following command - sudo virt-manager Download ubuntu-20.04.3-desktop-amd64.iso Start Virtual Machine Manager and install ubuntu. Once the inner VM is started run the below commands in the inner VM sudo apt install cpuid

Run below to get output of part 1:
cpuid -l 0x4FFFFFFF

Run below to get output of part 2:
cpuid -l 0x4FFFFFFE

Assignment 3

Answer 1:

Archana Miyar Kamath (015276378)
Setting up Virtual machine
Code changes in cpuid.c and vmx.c.
Building the code changes.
Debugging the build issues.

Mounica Kamireddy (015949414)
Understanding the usage of int, atomic variables and u32, u64 etc
Installing nested machine and implemented test cases.
Debugging issues while installing nested virtual machine.

Answer 2

Part 1:

For CPUID leaf node %eax=0x4FFFFFFD: Return the number of exits for the exit number provided (on input) in %ecx This value should be returned in %eax

Part 2:

For CPUID leaf node %eax=0x4FFFFFFC: Return the time spent processing the exit number provided (on input) in %ecx Return the high 32 bits of the total time spent for that exit in %ebx

Answer 2:

Repeat the steps from assignment 2 to build the changes. As follows

Do the code changes in cpuid.c and vmx.c path : /linux/arch/x86/kvm/cpuid.c /linux/arch/x86/kvm/vmx/vmx.c

Run the below commands in order

sudo make modules sudo make INSTALL_MOD_STRIP=1 modules_install sudo make install

lsmod | grep kvm // These below two files should not be there, if so remove them sudo rmmod kvm_intel sudo rmmod kvm

lsmod | grep kvm (kvm_intel and kvm should not be in response)

sudo modprobe kvm sudo modprobe kvm_intel

Reboot the VM

Test in inner VM

Run below to get output of part 1 cpuid -l 0x4FFFFFFD -s 0

Run below to get output of part 2 cpuid -l 0x4FFFFFFC -s 0

Execute dmesg in outer VM Link to output - Output

Answer 3

Comment on the frequency of exits – does the number of exits increase at a stable rate? Or are there more exits performed during certain VM operations? Approximately how many exits does a full VM boot entail?

Frequency of some exits have increased, while some remain almost same. Exit 0 (Exception or non-maskable interrupt) does not show any notable difference (Changed from 8919 to 8945) but Exit 1(External interrupt) almost doubles (From 27140 to 56295). I opened firefox and searched for some terms in the inner VM to note the difference in number of exits. 46 (Access to GDTR or IDTR), 48 (EPT violation) 49 (EPT misconfiguration) have increased a lot, where as reason code 28 to 32 does not show any much difference. Most of the other exit code shows 0, that is exit did not occur at all.

The total number of exits on inner VM reboot, as follows

Before count 001f17e3 = 2037731 decimal

After count 0030b376 = 3191670 decimal

Difference is 11,53,939

Answer 4

Of the exit types defined in the SDM, which are the most frequent? Least?

Most frequent Exit : 0 Exception or non-maskable interrupt (NMI)
Exit : 1 External interrupt
Exit : 7 Interrupt window
Exit : 10 CPUID
Exit : 12 HLT
Exit : 28 Control-register accesses
Exit : 30 I/O instruction
Exit : 31 RDMSR
Exit : 32 WRMSR
Exit : 48 EPT violation
Exit : 49 EPT misconfiguration

Least frequent
Exit : 29 MOV DR
Exit : 46 Access to GDTR or IDTR.
Exit : 54 WBINVD or WBNOINVD
Exit : 55 XSETBV

Rest of the exits did not occur.

Assignment 4

Answer 1:

Archana Miyar Kamath (015276378)
Executed the changes required for assignment 4.
Captured the resulted.

Mounica Kamireddy (015949414)
Comparison of results (with ept and without ept)
Noting the difference and coming to a conclusion.

Answer 2

with ept - with_ept

without ept - without_ept

Answer 3

The result was expected. The total exits have increased. The inner VM started with more delay with shadow paging when compared to boot time with nested paging. The exits that were zero with nested paging remained zero with shadow paging (except Exit:14, Exit 33, Exit 58)

Exit 14: INVLPG
count - 163046

Exit 33: VM-entry failure due to invalid guest state
count - 22789

Exit 58: INVPCID
count - 871

Answer 4

With ept(Nested Paging)

Relatively faster boot time for inner vm.
Exit count for reason 14, 33 and 58 were zero.
The exits count of other reason codes were lower than that of shadow paging.

Without ept(Shadow Paging)

Relatively slower boot time for inner vm.
Exit count for reason 14, 33 and 58 were not zero.
The exits count of other reason codes were higher than that of nested paging.