A systems software double-header: Surge and GOTO

I recently returned from a systems software double-header, presenting at two of the industry’s best conferences: Surge in Baltimore and GOTO in Aarhus, Denmark. These conferences have much in common: they are both expertly run; they both seek out top technical content; they both attract top technologists; and (it must be said) they both take excellent care of their speakers!

At Surge, I presented with Brendan Gregg on DIRT in production. This was a dense presentation, but it was fun for us to recount a sampling of the kinds of issues that we have seen in the new breed latency-sensitive data-intensive application. I think Brendan and I both agreed that this presentation could have easily been two or three times the length; we have plenty of scars from finding latency outliers in DIRTy systems! Beyond presenting, the lightning talks were a highlight for me at this year’s Surge. I am an inveterate disaster porn addict, so it should come as no surprise that I particularly enjoyed Chris Burroughs’ retelling of how his day that he had set aside to prepare his lightning talk was beset by disaster (including a PDU failure at the moment that it seemed things couldn’t get any worse), and then Scott Sanders on the surprisingly nasty failure modes that one can see in a mobile app when the root-cause is a botched datacenter upgrade in your mobile provider. For my own lightning talk, I dug into the perverse history of what may well be the most obscure two-letter Unix command: the “old and rarely used” ta; hopefully others enjoyed learning about the history of this odd little wart as much as I did!

Update: This lightning talk is now online and can be viewed here.

Dave and me with Lars Bak, Kasper Lund and the Dart (né V8) team

After Surge, Dave and I headed east, to Denmark. There, we presented on dynamic languages in production. Our purposes for going to Aarhus were more than just GOTO, however: we also went to meet with Lars Bak, Kasper Lund, Erik Corry, the inimitable Mr. Aleph and the other engineers behind V8. As the birthplace of V8, Aarhus has always had a Dagobah-like appeal to me: I feel that when on the freezing ice planet that is mdb_v8.c, Dave and I might as well have seen Obi-Wan Kenobi as an apparition, telling us we must go to Aarhus if we wanted to truly understand objects-inl.h. Dave and I immensely enjoyed our time with Lars and his team; it is rare to be in the presence of a team that has single-handedly changed the trajectory of software – and it is rarer still to be able bring one’s own technology to such an august table. We were thrilled to be able to demonstrate the confluence of our technology (specifically, MDB and DTrace) with theirs – and to brainstorm ways that we might better solve the abstract problem of production debuggability of dynamic environments. This is a problem that we’ve been after in one way or another for nearly a decade, but this was the first time that we were having the conversation early enough in the VM implementation phase (specifically, of Dart) that one can hope to get ahead of the problem instead of chasing it. In particular, Lars’s team has a very interesting idea for debuggability of Dart that – if successful – may allow it to achieve our nirvana of total postmortem debuggability in a dynamic environment. That, coupled with its enlightened (C-based!) native interface, left Dave and me optimistic about the long-term prospects of Dart on the server side. It’s obviously early days still for Dart, but given the pedigree and talent of the team and their disposition towards making it both highly performing and highly debuggable, it’s clearly something to pay attention to on the server side.

Presentation and meeting accomplished, I still had a burning question: what was all of this terrific VM talent doing in the second city of a small country with an excruciatingly difficult language to pronounce? (Aside: ask a Dane to say “dead red red-eyed rotten smoked trout” in Danish.) In talking to Lars, Kasper and Erik about how they ended up in Aarhus a common theme emerged: all had – in one way or another – been drawn to work with Professor Ole Lehrmann Madsen on the VM for the BETA language. (No, I hadn’t heard of BETA either, and yes, it’s in all caps – the name is apparently meant to be screamed.) Everyone I spoke with had such fond memories of BETA (“you know, it was actually a very interesting system…”) that I have taken to calling this school of VM engineers the BETA School. The work and influence of the engineers of the BETA School extend beyond Google, with its members having done foundational VM work at VMware, Sun and a slew of startups.

I suppose I shouldn’t be surprised that this cluster of engineers has a professor and a university at its root; Brown’s Tom Doeppner has served a similar role for us, with his CS169 giving rise to an entire generation of OS innovation. The two schools of engineers share much in common: both are highly technical and close-knit – but with a history of being open to newcomers and an emphasis on developing their junior members. (Of course, the difference is that we of the CS169 School don’t all live in Providence.) And the intersection of the two schools is particularly interesting: VMware’s seminal ASPLOS paper is a joint product of Keith Adams of the CS169 School and Ole Agesen of the BETA School.

Curiosity about Aarhus and its origins sated, I arrived home from my systems software double header exhausted but also inspired: there seems to be no limit of hard, interesting problems in our domain – and many profoundly talented engineers to tackle them!