Multi-Hypervisor Virtual Machines

Traditional system virtual machines (VMs) are controlled by only one hypervisor at a time. Consequently, a VM is limited to the set of hypervisor-level services provided by the sole controlling hypervisor. We propose the notion of a multi-hypervisor virtual machine, or Span VM, that is simultaneously controlled by multiple co-located hypervisors. A Span VM runs an unmodified guest operating system and benefits from different services offered by the underlying hypervisors. Using nested virtualization, we demonstrate how two or more hypervisors can cooperatively exert control over a Span VM's memory, VCPUs, and I/O devices. A rich set of hypervisor-level services have been recently proposed, such as guest monitoring, rootkit detection, high availability, and live guest patching. We envision Span VMs to enable a new hypervisor ecosystem in which such services are provided by specialized hypervisors that augment the base hypervisor. We have designed and implemented a prototype systems support for Span VMs using the KVM/QEMU platform. Our current prototype can support Span VMs that use multiple hypervisor services such as VM introspection, network monitoring, and guest mirroring, with performance comparable to traditional nested VMs.


  1. Kartik Gopalan, Rohith K. Raghavendra, Hardik Bagdi, Yaohui Hu, Dan Williams, Nilton Bila, Multi-Hypervisor Virtual Machines: Enabling an Ecosystem of Hypervisor-level Services, In USENIX ATC 2017, Santa Clara, CA, USA. [pdf] [bibtex]
  2. Dan Williams, Yaohui Hu, Umesh Deshpande, Nilton Bila, Kartik Gopalan, Hani Jamjoom, Enabling Efficient Hypervisor-as-a-Service Clouds with Ephemeral Virtualization, In International Conference on Virtual Execution Environments (VEE) 2016. [pdf] [bibtex]
  3. Hardik Bagdi, Rohith Kugve, and Kartik Gopalan, HyperFresh: Live Refresh of Hypervisors Using Nested Virtualization,, In 8th ACM/SIGOPS Asia Pacific Workshop on Systems (APSys), 2017, Mumbai, India. [pdf] [bibtex]
  4. Ozgur Kilic, Spoorti Doddamani, Aprameya Bhat, Hardik Bagdi, Kartik Gopalan, Overcoming virtualization overheads in large VCPU virtual machines., Accepted for publication In MASCOTS 2018, Miwaukee, WI. [pdf] [bibtex]
  5. SOCC 2016 Poster
  6. NSF PI Meeting Poster
  7. NSF Award