The Observation Deck

KVM on illumos

August 15, 2011

A little over a year ago, I came to Joyent because of a shared belief that systems software innovation matters — that there is no level of the software stack that should be considered off-limits to innovation. Specifically, we ardently believe in innovating in the operating system itself — that extending and leveraging technologies like OS virtualization, DTrace and ZFS can allow one to build unique, differentiated value further up the software stack. My arrival at Joyent also coincided with the birth of illumos, an effort that Joyent was thrilled to support, as we saw that rephrasing our SmartOS as an illumos derivative would allow us to engage in a truly open community with whom we could collaborate and collectively innovate.

In terms of the OS technology itself, as powerful as OS virtualization, DTrace and ZFS are, there was clearly a missing piece to the puzzle: hardware virtualization. In particular, with advances in microprocessor technology over the last five years (and specifically with the introduction of technologies like Intel’s VT-x), one could (in principle, anyway) much more easily build highly performing hardware virtualization. The technology was out there: Fabrice Bellard‘s cross-platform QEMU had been extended to utilize the new microprocessor support by Avi Kivity and his team with their Linux kernel virtual machine (KVM).

In the fall of last year, the imperative became clear: we needed to port KVM to SmartOS. This notion is almost an oxymoron, as KVM is not “portable” in any traditional sense: it interfaces between QEMU, Linux and the hardware with (rightful) disregard for other systems. And as KVM makes heavy use of Linux-specific facilities that don’t necessarily have clean analogs in other systems, the only way to fully scope the work was to start it in earnest. However, we knew that just getting it to compile in any meaningful form would take significant effort, and with our team still ramping up and with lots of work to do elsewhere, the reality was that this would have to start as a small project. Fortunately, Joyent kernel internals veteran Max Bruning was up to the task, and late last year, he copied the KVM bits from the stable Linux 2.6.34 source, took out a sharpened machete, and started whacking back jungle…

Through the winter holidays and early spring (and when he wasn’t dispatched on other urgent tasks), Max made steady progress — but it was clear that the problem was even more grueling than we had anticipated: because KVM is so tightly integrated into the Linux kernel, it was difficult to determine dependencies — and hard to know when to pull in the Linux implementation of a particular facility or function versus writing our own or making a call to the illumos equivalent (if any). By the middle of March, we were able to execute three instructions in a virtual machine before taking an unresolved EPT violation. To the uninitiated, this might sound pathetic — but it meant that a tremendous amount of software and hardware were working properly. Shortly thereafter, with the EPT problem fixed, the virtual machine would happily run indefinitely — but clearly not making any progress booting. But with this milestone, Max had gotten to the point where the work could be more realistically parallelized, and we had gotten to the point as a team and a product where we could dedicate more engineers to this work. In short, it was time to add to the KVM effort — and Robert Mustacchi and I roped up and joined Max in the depths of KVM.

With three of us on the project and with the project now more parallelizable, the pace accelerated, and in particular we could focus on tooling: we added an mdb dmod to help us debug, 16-bit disassembler support that was essential for us to crack the case on the looping in real-mode, and DTrace support for reading the VMCS. (And we even learned some surprising things about GNU C along the way!) On April 18th, we finally broke through into daylight: we were able to boot into KMDB running on a SmartOS guest! The pace quickened again, as we now had the ability to interactively ascertain state as seen by the guest and much more quickly debug issues. Over the next month (and after many intensive debugging sessions), we were able to boot Windows, Linux, FreeBSD, Haiku, QNX, Plan 9 and every other OS we could get our hands on; by May 13th, we were ready to send the company-wide “hello world” mail that has been the hallmark of bringup teams since time immemorial.

The wind was now firmly at our backs, but there was still much work to be done: we needed to get configurations working with large numbers of VCPUs, with many VMs, over long periods of time and significant stress and so on. Over the course of the next few months, the bugs became rarer and rarer (if nastier!), and we were able to validate that performance was on par with Linux KVM — to the point that today we feel confident releasing illumos KVM to the greater community!

In terms of starting points, if you just want to take it for a spin, grab a SmartOS live image ISO. If you’re interested in the source, check out the illumos-kvm github repo for the KVM driver itself, the illumos-kvm-cmd github repo for our QEMU 0.14.1 tree, and illumos itself for some of our KVM-specific tooling.

One final point of validation: when we went to integrate these changes into illumos, we wanted to be sure that they built cleanly in a non-SmartOS (that is, stock illumos) environment. However, we couldn’t dedicate a machine for the activity, so we (of course) installed OpenIndiana running as a KVM guest on SmartOS and cranked the build to verify our KVM deltas! As engineers, we all endeavor to build useful things, and with KVM on illumos, we can assert with absolute confidence that we’ve hit that mark!

44 Responses

  1. Congratulations!
    Glad to see that the Illumos project is carrying on (Sun) Solaris’s legacy as a nexus of systems innovation.

    I see that kvm (and qemu) are licensed under the GPL. Does the fact that illumos-kvm is a loadable kernel module (and not part of the illumos kernel itself) immunize it to the viral nature of the GPL?

    Great work, I’ll give it a spin soon!

    1. Nick,

      Thanks for the kind words! And no, there are no licensing issues here: KVM is GPL’d, but the kernel is no way a derived work of it. Looking forward to you giving it a spin!

      1. The KVM GPL license doesn’t change the license of the kernel.

        However, when you distribute the KVM module combined with the kernel, the KVM license says the combined work has to be distributed under the GPL (or compatible) license or you don’t have the right to distribute the GPL-licensed work.

    1. Migration is much more a QEMU-level issue than a KVM-level issue. Looking into the future, we do intend to integrate QEMU migration with ZFS remote replication; stay tuned.

    1. Yes, stay tuned — we’re going to be fleshing out SmartOS with enough components of our larger SmartDataCenter systems to (for example) be able to provision a zone with a KVM instance.

  2. Fantastic work! I’m thrilled to see that several of you who made a real difference to Solaris before the acquisition is providing new innovations to illumos.

    I’m working with large scale deployment of zones myself so I am especially thrilled to see that KVM can with benefit be run inside of a zone. The added security comes at a good time after the “breaking out of KVM” presentation at Blackhat (ironically by a Oracle employee).

  3. Hi, Brian:

    Congrats on the release. Two things:

    1. Can you clarify if Joyent SmartOS now is Illumos only? I mean is it 100% clean from Oracle’s closed code base?

    2. Can Joyent release a SmartOS operating system user guide pdf file? It took me nearly 10 minutes to find admin/admin and root/root are the login passwd. And I haven’t found the commands to do a HDD install from the livecd.iso released today.

    Thanks for the hard work and contribution to Illumos base.

    1. I would start with Ryan’s guide, but yes, we know that this is — in the immortal words of Bart Smaalders — all blade and no handle. We have lots of planned improvements coming; thank you for your words of encouragement, and stay tuned!

  4. What about the GPL? Isn’t that an issue that you’re taking code from inside the Linux kernel to integrate to a kernel that is not distributed under the GPL? I know Linux was never able to integrate ZFS because of the license disparity (in the other direction, distributing ZFS under the GPL would not be possible). How were you able to work around this issue?

  5. Fantastic news, I use VirtualBox right now but seeing that is is now somewhat closed off by Oracle and despite several months of efforts I can not even purchase licences or support for it (Oracle doesn’t understand how to quote software support or licenses).

    The ability to switch to something else that is “native” illumOS is happy news indeed.

    I look forward to the ability to live migrations. I will be giving this a spin in a few days and look forward to see if it can import OVF, VDI or VMDK images.

    1. Jon, Yep — it’s all coming. Also — and at the risk of upselling you 😉 — you might be interested in our SmartDataCenter product, which gives you the ability to deploy, manage and observe a cloud of SmartOS-based compute nodes. (Our 6.5 version of the product (due out in early October) will have KVM support.)

  6. Where is the documentation?

    pkgtrans(1M) is present, but pkgadd(1M) and /var/sadm/install/contents is missing.

    What is used as a software management subsystem?

  7. In the ensuing excitement, I almost forgot: congratulations!

    When you wrote “we finally broke through into daylight” it was a reminder of the engineering toil (and excitement!) that we as a team are also going through; at that moment, I knew exactly how that must have felt.

    You did what others only tried to do.

  8. Pingback: SmartOS
  9. I tried your SmartOS live disc here as a trial on my system. It booted up, but claims that there’s no support in my hardware for KVM. I run KVM on it though linux on this hardware all the time. Have you tested this at all with AMD64 based chips, or was all of the work done with Intel chips?

    1. HI,

      it hung at when booting up, at “stage 2 …” for me ..
      what are the hw pre-req for running the live/ or installations ?

      thanks, and this is a really great step forward!

Leave a Reply

Recent Posts

November 18, 2023
November 27, 2022
October 11, 2020
July 31, 2019
December 16, 2018
September 18, 2018
December 21, 2016
September 30, 2016
September 26, 2016
September 13, 2016
July 29, 2016
December 17, 2015
September 16, 2015
January 6, 2015
November 10, 2013
September 3, 2013
June 7, 2012
September 15, 2011
August 15, 2011
March 9, 2011
September 24, 2010
August 11, 2010
July 30, 2010
July 25, 2010
March 10, 2010
November 26, 2009
February 19, 2009
February 2, 2009
November 10, 2008
November 3, 2008
September 3, 2008
July 18, 2008
June 30, 2008
May 31, 2008
March 16, 2008
December 18, 2007
December 5, 2007
November 11, 2007
November 8, 2007
September 6, 2007
August 21, 2007
August 2, 2007
July 11, 2007
May 20, 2007
March 19, 2007
October 12, 2006
August 17, 2006
August 7, 2006
May 1, 2006
December 13, 2005
November 16, 2005
September 13, 2005
September 9, 2005
August 21, 2005
August 16, 2005