Hubris and Humility

When we started Oxide, we knew we were going to take a fresh look at the entire system. We knew, for example, that we wanted to have a true hardware root of trust and that we wanted to revisit the traditional BMC. We knew, too, that we would have our own system software on each of these embedded systems, and assumed that we would use an existing open source operating system as a foundation. However, as we waded deeper into these embedded systems and especially their constraints of security and robustness we found that what we wanted out of the existing operating systems wasn’t necessarily what they offered (or even wanted to offer).

As time went on in early 2020 and we found ourselves increasingly forcing existing systems out of the comfort of their design centers, we wondered: was our assumption of using an existing system wrong? Should we in fact be exploring our own de novo operating system? In particular, our colleague Cliff Biffle, who had a ton of experience with both Rust and embedded systems, had a vision for what such a system might look like (namely, the system that he had always wanted for himself!). Cliff dove into a sketch of his ideas, giving the nascent system a name that felt perfectly apt: Hubris.

After just a few weeks, Cliff’s ideas were taking concrete shape, and it was clear that there was a lot to like: the emerging Hubris was an all-Rust system that was not distracting itself by accommodating other runtimes; it was microkernel-based, allowing for safety and isolation; it employed a strictly synchronous task model, allowing for it be easily developed and comprehended; and it was small and light, allowing it to fit into some of the tight spots we envisioned for it. But for me, it was the way Cliff thought about the building of the system that really set Hubris apart: instead of having an operating system that knows how to dynamically create tasks at run-time (itself a hallmark of multiprogrammed, general purpose systems), Cliff had designed Hubris to fully specify the tasks for a particular application at build time, with the build system then combining the kernel with the selected tasks to yield a single (attestable!) image.

This is the best of both worlds: it is at once dynamic and general purpose with respect to what the system can run, but also entirely static in terms of the binary payload of a particular application — and broadly static in terms of its execution. Dynamic resource exhaustion is the root of many problems in embedded systems; having the system know a priori all of the tasks that it will ever see liberates it from not just a major source of dynamic allocation, but also from the concomitant failure modes. For example, in Hubris, tasks can always be safely restarted, because we know that the resources associated with a task are available if that task itself has faulted! And this eliminates failure modes in which dynamic task creation in response to load induces resource exhaustion; as Cliff has quipped, it is hard to have a fork bomb when the system lacks fork itself!

Precedence for the Hubris approach can be found in other systems like library operating systems, but there is an essential difference: Hubris is a memory-protected system, with tasks, the kernel, and drivers all in disjoint protection domains. (And yes, even in memory safe languages like Rust, memory protection is essential!) In this regard, Hubris represents what we like about Rust, too: a creative solution that cuts through a false dichotomy to yield a system that is at once nimble and rigorous.

It was clear that Cliff was on the right track with Hubris, and the rest of us jumped in with gusto. For my own part, with debugging and debuggability deep in my marrow, I set to work writing the debugger that I felt that we would need — and that Hubris deserved. Following Cliff’s lead, I dubbed it Humility, and it’s been exciting for me to see the debugger and the operating system work together to yield a higher-quality system.

We have known for quite some time that we would open source Hubris: not only is open source core to our own commercial thesis at Oxide, we also believe that the open source revolution — and its many advantages for customers — are long overdue in the lowest layers of the software stack. So we were waiting for the right occasion, and the Open Source Firmware Conference afforded us an excellent one: if you are a listener of our On the Metal podcast, you heard us talk about OSFC a bunch, and it felt entirely fitting that we would kickoff our own open source firmware contribution there. And while the conference starts today, the good news is that you haven’t missed anything! Or at least, not yet: the conference is virtual, so if you want to hear Cliff talk about Hubris in his own words — and it’s before 12:10 Pacific today — it’s not too late to buy a ticket! (The recording will naturally be released after the conference.)

And of course, if you just want to play with the system itself, Hubris and Humility are both now open source! Start by checking out the Hubris repo, the Hubris docs, and the Humility repo. (And if you are looking for a hardware vehicle for your exploration, take a look at the ST Nucleo-H753ZI evaluation board — a pretty capable computer for less than thirty bucks!) We believe Rust to be a part of the coming firmware revolution, and it’s exciting for us to have a system out there that embodies that belief!