A little over a year ago, I came to Joyent because of a shared belief that systems software innovation matters — that there is no level of the software stack that should be considered off-limits to innovation. Specifically, we ardently believe in innovating in the operating system itself — that extending and leveraging technologies like OS virtualization, DTrace and ZFS can allow one to build unique, differentiated value further up the software stack. My arrival at Joyent also coincided with the birth of illumos, an effort that Joyent was thrilled to support, as we saw that rephrasing our SmartOS as an illumos derivative would allow us to engage in a truly open community with whom we could collaborate and collectively innovate.
In terms of the OS technology itself, as powerful as OS virtualization, DTrace and ZFS are, there was clearly a missing piece to the puzzle: hardware virtualization. In particular, with advances in microprocessor technology over the last five years (and specifically with the introduction of technologies like Intel’s VT-x), one could (in principle, anyway) much more easily build highly performing hardware virtualization. The technology was out there: Fabrice Bellard‘s cross-platform QEMU had been extended to utilize the new microprocessor support by Avi Kivity and his team with their Linux kernel virtual machine (KVM).
In the fall of last year, the imperative became clear: we needed to port KVM to SmartOS. This notion is almost an oxymoron, as KVM is not “portable” in any traditional sense: it interfaces between QEMU, Linux and the hardware with (rightful) disregard for other systems. And as KVM makes heavy use of Linux-specific facilities that don’t necessarily have clean analogs in other systems, the only way to fully scope the work was to start it in earnest. However, we knew that just getting it to compile in any meaningful form would take significant effort, and with our team still ramping up and with lots of work to do elsewhere, the reality was that this would have to start as a small project. Fortunately, Joyent kernel internals veteran Max Bruning was up to the task, and late last year, he copied the KVM bits from the stable Linux 2.6.34 source, took out a sharpened machete, and started whacking back jungle…
Through the winter holidays and early spring (and when he wasn’t dispatched on other urgent tasks), Max made steady progress — but it was clear that the problem was even more grueling than we had anticipated: because KVM is so tightly integrated into the Linux kernel, it was difficult to determine dependencies — and hard to know when to pull in the Linux implementation of a particular facility or function versus writing our own or making a call to the illumos equivalent (if any). By the middle of March, we were able to execute three instructions in a virtual machine before taking an unresolved EPT violation. To the uninitiated, this might sound pathetic — but it meant that a tremendous amount of software and hardware were working properly. Shortly thereafter, with the EPT problem fixed, the virtual machine would happily run indefinitely — but clearly not making any progress booting. But with this milestone, Max had gotten to the point where the work could be more realistically parallelized, and we had gotten to the point as a team and a product where we could dedicate more engineers to this work. In short, it was time to add to the KVM effort — and Robert Mustacchi and I roped up and joined Max in the depths of KVM.
With three of us on the project and with the project now more parallelizable, the pace accelerated, and in particular we could focus on tooling: we added an mdb dmod to help us debug, 16-bit disassembler support that was essential for us to crack the case on the looping in real-mode, and DTrace support for reading the VMCS. (And we even learned some surprising things about GNU C along the way!) On April 18th, we finally broke through into daylight: we were able to boot into KMDB running on a SmartOS guest! The pace quickened again, as we now had the ability to interactively ascertain state as seen by the guest and much more quickly debug issues. Over the next month (and after many intensive debugging sessions), we were able to boot Windows, Linux, FreeBSD, Haiku, QNX, Plan 9 and every other OS we could get our hands on; by May 13th, we were ready to send the company-wide “hello world” mail that has been the hallmark of bringup teams since time immemorial.
The wind was now firmly at our backs, but there was still much work to be done: we needed to get configurations working with large numbers of VCPUs, with many VMs, over long periods of time and significant stress and so on. Over the course of the next few months, the bugs became rarer and rarer (if nastier!), and we were able to validate that performance was on par with Linux KVM — to the point that today we feel confident releasing illumos KVM to the greater community!
In terms of starting points, if you just want to take it for a spin, grab a SmartOS live image ISO. If you’re interested in the source, check out the illumos-kvm github repo for the KVM driver itself, the illumos-kvm-cmd github repo for our QEMU 0.14.1 tree, and illumos itself for some of our KVM-specific tooling.
One final point of validation: when we went to integrate these changes into illumos, we wanted to be sure that they built cleanly in a non-SmartOS (that is, stock illumos) environment. However, we couldn’t dedicate a machine for the activity, so we (of course) installed OpenIndiana running as a KVM guest on SmartOS and cranked the build to verify our KVM deltas! As engineers, we all endeavor to build useful things, and with KVM on illumos, we can assert with absolute confidence that we’ve hit that mark!
44 Responses
From all of the OpenIndiana team, I’d like to offer a heartfelt thank you to Joyent! Bringing KVM to illumos brings yet another great technology to the stack and helps demonstrate the commitment of those behind it.
WOW WOW WOW – GREAT – WONDERFUL – WOW.
YOU’RE WONDERFUL really …
thanks a lot the community will thank you all a lot – really really really !
Michael
Congratulations!
Glad to see that the Illumos project is carrying on (Sun) Solaris’s legacy as a nexus of systems innovation.
I see that kvm (and qemu) are licensed under the GPL. Does the fact that illumos-kvm is a loadable kernel module (and not part of the illumos kernel itself) immunize it to the viral nature of the GPL?
Great work, I’ll give it a spin soon!
Nick,
Thanks for the kind words! And no, there are no licensing issues here: KVM is GPL’d, but the kernel is no way a derived work of it. Looking forward to you giving it a spin!
The KVM GPL license doesn’t change the license of the kernel.
However, when you distribute the KVM module combined with the kernel, the KVM license says the combined work has to be distributed under the GPL (or compatible) license or you don’t have the right to distribute the GPL-licensed work.
Woot!
Thanks!
Does it handle live migrations (aka vmotion)? If so, what’s your storage story.
Migration is much more a QEMU-level issue than a KVM-level issue. Looking into the future, we do intend to integrate QEMU migration with ZFS remote replication; stay tuned.
It’s Haiku, not HaikuOS
Updated — thank you! (And have I mentioned the strange affection that I have developed for Haiku over the course of this project?)
Is it because Haiku is based upon BeOS, which in turn draws from AmigaOS?
Guys, good work!, even the story was catching… I see a bright future ahead!
Fantastic work!
Will you pursue writing a qemu-equivalent frontend which can/will better integrate with things like zonecfg and friends?
Yes, stay tuned — we’re going to be fleshing out SmartOS with enough components of our larger SmartDataCenter systems to (for example) be able to provision a zone with a KVM instance.
Fantastic work! I’m thrilled to see that several of you who made a real difference to Solaris before the acquisition is providing new innovations to illumos.
I’m working with large scale deployment of zones myself so I am especially thrilled to see that KVM can with benefit be run inside of a zone. The added security comes at a good time after the “breaking out of KVM” presentation at Blackhat (ironically by a Oracle employee).
This is like Christmas.
Dave, It’s our privilege to play the role of Santa. 😉
Hi, Brian:
Congrats on the release. Two things:
1. Can you clarify if Joyent SmartOS now is Illumos only? I mean is it 100% clean from Oracle’s closed code base?
2. Can Joyent release a SmartOS operating system user guide pdf file? It took me nearly 10 minutes to find admin/admin and root/root are the login passwd. And I haven’t found the commands to do a HDD install from the livecd.iso released today.
Thanks for the hard work and contribution to Illumos base.
TS –
This guide (by Ryan Nelson of Joyent) should help in the mean time:
http://www.ryan.net/smartos-disk-blogpost/real_disk_smartos.html
I would start with Ryan’s guide, but yes, we know that this is — in the immortal words of Bart Smaalders — all blade and no handle. We have lots of planned improvements coming; thank you for your words of encouragement, and stay tuned!
Better than christmas
This is really great news, but … how do I install smartos to hd.
What about the GPL? Isn’t that an issue that you’re taking code from inside the Linux kernel to integrate to a kernel that is not distributed under the GPL? I know Linux was never able to integrate ZFS because of the license disparity (in the other direction, distributing ZFS under the GPL would not be possible). How were you able to work around this issue?
No, our KVM port remains GPL and its own work (and lives in its own repo) — the illumos kernel is CDDL but is in no way a derived work of our KVM port.
Fantastic news, I use VirtualBox right now but seeing that is is now somewhat closed off by Oracle and despite several months of efforts I can not even purchase licences or support for it (Oracle doesn’t understand how to quote software support or licenses).
The ability to switch to something else that is “native” illumOS is happy news indeed.
I look forward to the ability to live migrations. I will be giving this a spin in a few days and look forward to see if it can import OVF, VDI or VMDK images.
Jon, Yep — it’s all coming. Also — and at the risk of upselling you 😉 — you might be interested in our SmartDataCenter product, which gives you the ability to deploy, manage and observe a cloud of SmartOS-based compute nodes. (Our 6.5 version of the product (due out in early October) will have KVM support.)
I remember discussing Qemu with you years ago. I’m somehow not surprised you managed to do this port. 🙂
Where is the documentation?
pkgtrans(1M) is present, but pkgadd(1M) and /var/sadm/install/contents is missing.
What is used as a software management subsystem?
In the ensuing excitement, I almost forgot: congratulations!
When you wrote “we finally broke through into daylight” it was a reminder of the engineering toil (and excitement!) that we as a team are also going through; at that moment, I knew exactly how that must have felt.
You did what others only tried to do.
Does KVM work under SPARC?
Will we be able to use KVM (QEMU) under SPARC to provide access to x86, x64, or ARM systems, possibly running Solaris?
Great stuff, Bryan. Great contribution to illumos and the industry.
You rock! Thanks for all your incredible efforts!
I tried your SmartOS live disc here as a trial on my system. It booted up, but claims that there’s no support in my hardware for KVM. I run KVM on it though linux on this hardware all the time. Have you tested this at all with AMD64 based chips, or was all of the work done with Intel chips?
HI,
it hung at when booting up, at “stage 2 …” for me ..
what are the hw pre-req for running the live/ or installations ?
thanks, and this is a really great step forward!
sam