The Observation Deck

The right place at the right time

Thu, 08 Jan 2026 00:00:00 +0000

Recently, the New Stack had a piece excerpting some of my conversation with Gergely Orosz. In introducing me in the piece, Joab Jackson writes "Cantrill has had a knack at being at the right place at the right time." While this is objectively true at some level, it also gave me a double take: it rarely felt true at the time.

When I entered university in 1992, it didn’t feel like the right time: the economy for new grads was very grim — and I knew plenty of folks who had done at well at top schools who were struggling to find work (and accepting part time jobs that didn’t need a college degree at all while they searched for something better). I never doubted going to school, but I also have never taken a job for granted.

When I fell in love with computer science as an undergraduate and realized that I wanted to become a software engineer, it didn’t feel like the right time: Ed Yourdon had just written The Decline and Fall of the American Programmer, which boldly told any young computer science student that they were wasting their time — that all programming jobs would be done by cheap labor abroad. This argument felt wrong, but I was too in love with computer science to be talked out of it anyway.

When I decided that I was specifically interested in operating systems kernel development, it definitely didn’t feel like the right time: the conventional wisdom in the mid-1990s was that operating systems were done — that Unix was in decline and that the future clearly belonged to Microsoft. I ardently disagreed with this, and my conviction brought me in 1996 to the one company that unequivocally shared it: Sun Microsystems.

When the internet exploded in the late-1990s, I was — at last — at the right place at the right time. But it was short-lived: when the Dot Com Boom turned to a Dot Com Bust, we were fighting conventional wisdom yet again. Didn’t the bust mean that there was no future to this stuff?

When we started Oxide in 2019, it was almost comforting when it didn’t feel like the right time: Sand Hill investors told us that "we only fund SaaS companies", that our $20M seed round was "too big", that "there is no market." (And most absurdly: "if this is such a good idea, why are you the only ones doing it?") Six years later — with VMware customers wanting to get away from Broadcom and with frontier AI companies realizing that there is (in fact!) a lot of general purpose CPU involved in their workloads — our timing looks perfect, but in fact we are just resilient.

So, my advice: have the courage to follow your heart — and disregard what others say about your timing. This is not an assurance of an easy path (to the contrary, going your own way will assuredly be a struggle!), but you will have the solace of knowing that you were true to yourself — that you did your best work. And who knows: maybe years down the stretch, someone will observe that you have the knack for the very thing that you disregarded…

Note: Also published as a LinkedIn post.

Love your customers

Wed, 31 Dec 2025 00:00:00 +0000

Note: This was originally published as a LinkedIn post on December 20, 2025.

There were many reasons why I left Oracle shortly after it acquired Sun, but first among them: I could not stomach the disdain that Oracle had for its own customers. Sun — for all of its (many!) other faults — loved its customers.

I bring this up because I was recently contacted by a former Sun colleague, who had some technical questions about Oxide. His career had taken him to VMware and now to Broadcom, where he has a senior engineering management position. After answering the questions, I asked him how things were going at Broadcom. He left Sun before the Oracle transition, but I could see potential cultural similarities in the Broadcom acquisition of VMware — and I imagine it could be jarring.

The response surprised me: "It’s much needed. Broadcom is a breath of fresh air." Now, sometimes an acquirer makes important decisions that employees feel are overdue. Still, though, I’ve talked to a lot of former VMware customers, and… "breath of fresh air" is decidedly not how they describe it. So I asked the obvious follow-up: "How are customers feeling about it?"

His response speaks for itself: "Actually the majority of them are extremely happy about the product changes. I think the transition to subscription and to core licensing is never easy but we got past that. We have customers signing very long contracts right now which means they are not looking to change. Btw some that tried came crawling back lol"

That last sentence — "some that tried came crawling back lol" — made my stomach turn, reminding me viscerally of the contempt that Oracle had for its own customers that I had found so loathsome. I was a bit stunned; had my former colleague taken nothing away from Sun with respect to earning customer trust?! And my response was arguably a bit flip: "Just don’t ask AT&T, Fidelity, Tesco and United Healthcare, I guess? Or do those get counted in the 'crawling back' category?" (Of note: these VMware customers have all litigated against Broadcom.)

His response was consistent with what he’d already revealed about his own character — but no less galling: "Yes actually they all did. Again litigation doesn’t mean they moved on."

So I guess it needs to be said: suing your customers is gross — and being sued by your customers is shameful. Companies that have disdain for their own customers will be reviled in return. Such companies may be able to thrive in the short term, but they do not endure in the limit. Certainly, these companies not endure as innovators: when coercion is your business model, innovation is not merely unnecessary but actively antithetical.

At Oxide, we believe that the best companies — the most enduring ones — love their customers, and we have made this explicit in our mission. Our love for our customers has any number of manifestations but not least: we believe in innovating on our customer’s behalf — in working hard to solve their thorniest problems. When we set out, we said that we aspired to be "the kind of company that customers will love to buy from — and employees will be proud to work for." Six years on, it a single source of pride for us that we are succeeding in both of these dimensions, with heartfelt thanks and deep appreciation (and happy holidays!) to our customers and team!

Your intellectual fly is open

Fri, 05 Dec 2025 00:00:00 +0000

Note: This was originally published as a LinkedIn post on November 11, 2025.

I need to make a painful confession: somehow, LinkedIn has become an important social network to me. This isn’t necessarily due to LinkedIn’s sparkling competence, of course. To the contrary, LinkedIn is the Gerald Ford of social networks: the normal one that was left standing as the Richard Nixons and the Spiro Agnews of social media imploded around them. As with Gerald Ford, with LinkedIn we know that we’re getting something a bit clumsy and boring, but (as with post-Watergate America!), we’re also getting something that isn’t totally crooked — and that’s actually a bit of a relief.

But because I am finding I am spending more time here, we need to have some real talk: too many of you are using LLMs to generate content. Now, this isn’t entirely your fault: as if LLMs weren’t tempting enough, LinkedIn itself is cheerfully (insistently!) offering to help you "rewrite it with AI." It seems so excited to help you out, why not let it chip in and ease your own burden?

Because holy hell, the writing sucks. It’s not that it’s mediocre (though certainly that!), it’s that it is so stylistically grating, riddled with emojis and single-sentence paragraphs and "it’s not just… but also" constructions and (yes!) em-dashes that some of us use naturally — but most don’t (or shouldn’t).

When you use an LLM to author a post, you may think you are generating plausible writing, but you aren’t: to anyone who has seen even a modicum of LLM-generated content (a rapidly expanding demographic!), the LLM tells are impossible to ignore. Bluntly, your intellectual fly is open: lots of people notice — but no one is pointing it out. And the problem isn’t merely embarrassment: when you — person whose perspective I want to hear! — are obviously using an LLM to write posts for you, I don’t know what’s real and what is in fact generated fanfic. You definitely don’t sound like you, so… is the actual content real? I mean, maybe? But also maybe not. Regardless, I stop reading — and so do lots of others.

To be clear, I think LLMs are incredibly useful: they are helpful for brainstorming, invaluable for comprehending text, and they make for astonishingly good editors. (And, unlike most good editors, you can freely ignore their well-meaning suggestions without fear of igniting a civil war over the Oxford comma or whatever.) But LLMs are also lousy writers and (most importantly!) they are not you. At best, they will wrap your otherwise real content in constructs that cause people to skim or otherwise stop reading; at worst, they will cause people who see it for what it is to question your authenticity entirely.

So please, if not for the sanity of all of us than just to give your own message the credit it deserves: have some confidence in your own voice — and write your own content.

A parade in Oakland

Sat, 04 Oct 2025 00:00:00 +0000

On a Sunday night not too long ago, something extraordinary happened in West Oakland: the Oakland Ballers — an independent baseball team not even two years old — won the Pioneer League championship, bringing a baseball championship to Oakland for the first time since 1989. The atmosphere was electric, but to an outsider, the roar of the packed house might have seemed perplexing: why such fervor for… an independent league baseball game?! The beauty of baseball is the narrative beyond the diamond, and that Sunday night in Oakland stands as an exemplar. Speaking personally, to understand not just why I was there with my wife and our teenage daughter, but the emotion behind our full-throated chants, you need to go back in time, to a little more than a decade ago.

It was 2014. After a promising season, the Oakland A’s had just had an early exit from the playoffs, having lost a gutting extra innings single-game wildcard in Kansas City. Our boys — then aged seven and ten — had had their hearts broken, in a way that a sports team uniquely can for kids of a certain age. The boys were weeping; we consoled them as parents of grieving sports fans do — not by saying that "it’s just a game" (an obvious lie!), but by saying what parents have said for generations: wait until next year. "Someday," I recall telling them as they went to bed, "perhaps decades from now, the A’s will be in the World Series in Oakland — and we will be there to watch it together." Tears dried, the boys went to sleep; there was, after all, always next year.

Fast forward a decade, to 2023, and I was being proven a liar — not by disappointment on the field, but by treachery off it: the owner of the Oakland A’s — titanic failson John Fisher — had won approval to move the team from Oakland; after years of threats, the A’s were actually leaving. As Brodie cried, so did we all. This emotion may seem confusing: is this grieving over… one less entertainment option? Well, no, it’s about more than the team or even the game — it’s about who we have watched those games with. It’s not just our fellow fans; it’s our friends — our families. In our house, it was our nuclear family: we had become A’s fans in 2012 when our oldest caught the baseball bug at seven (aside: he still has it). For other A’s fans, it was multi-generational; listen to Roberto Santiago describe it when he was interviewed at the Reverse Boycott at the Coliseum:

My dad didn’t know what to do with me, so he brought me here and we used to walk across that BART bridge to games, and it was like magic. I watched Dwayne Murphy and Ricky Henderson out there running the outfield, stealing bases, hats flying off, it was beautiful. And that’s what made me fall in love with baseball.

And then I had a little brother who is 13 years younger than me, and I didn’t know what to do with him. So what did I do? I brought him here. I brought him to baseball games, I brought him to Opening Day. His birthday is in April, and we would come to Opening Day every year. And when I went away to college, I would send him tickets to Opening Day every year. And he would call me for the national anthem, and he would call me for the seventh-inning stretch, and he would call me for Jason Giambi’s first at bat as a Yankee, so I could hear the boos.

And then when I had kids, I knew what to do with them. And so what did I do? I brought them here. My kids are so at home here, they come in here and they take off their jackets and they drop them in a corner and they run off because they feel like they’re at home. That’s what baseball in Oakland is. This is generations.

To rip a team out of a city, then, is to sour some of our dearest memories, ones that cannot be recreated; it is to come into a home and desecrate the family photo album. And the scar is permanent: look at the Sonics or the North Stars or the Expos or the Nordiques — to say nothing of the generation-defining trauma of the Colts or the Dodgers. For each of these — the most recent of which was nearly two decades ago — you won’t have to look long to find people who got passed it, but never, in fact, got over it.

And this is where I was when 2024 rolled around: passed it, maybe, but absolutely not over it.

And then, in March, a surprise. From the arsonous ashes of Oakland baseball emerged an unexpected green shoot: a new independent league team, the Oakland Ballers. In the video announcing the team, the words of Mistah F.A.B. echoed: "Anytime something ends, something else begins." I was elated at the prospect of the Ballers — and then doubly so when team co-founders Paul Freedman and Bryan Carmel joined us on Oxide and Friends. Somehow, in the span of just a few frenetic weeks in the spring of 2024, they turned historic Raimondi Park into a proper ballpark. The home opener in 2024 was sold out and while (per grand tradition of Oakland baseball!) the home team lost, the game was a blast.

As the 2024 season went on, the games felt like a neighborhood block party — you always saw people you knew. And the team knew their town: when they unveiled their new mascot, Scrappy the Possum, it was a pitch-perfect Gritty-esque reflection of the Oakland zeitgeist. The team made the playoffs that year, winning a game at home but losing in the semi-finals. I was at the game with the kids — and as my then-17-year-old said, "it’s great to have postseason baseball back in Oakland." And it was!

Then, the 2025 season — this season. Early in the season, there was a discernible shift. Something to know about the Pioneer League is that it’s professional, but it’s deliberately a development league: players can’t have more than three years of professional experience. And with the MLB draft having been drastically cut (from 40 rounds to 20), MLB-affiliated development leagues like the Pioneer League are increasingly where young players go to try to chase their dream of making it to the bigs. The Ballers were making a name for themselves among college players from the region — and the complexion of the roster was changing, with increasingly familiar names. These were kids who had grown up in Berkeley and Martinez and Hayward and Stockton — who had gone to JUCO at Chabot and Ohlone, to college at East Bay and Davis and Cal and St. Mary’s. These were… NorCal kids. Our kids. And they were good. Like, really good.

The structure of the postseaon in the Pioneer League is that the season is divided into two halves, with the top two teams of each half making the playoffs. This makes it exciting for the players and the fans: no matter how rocky a start a team has, they could still make the postseason with a strong second-half showing. But the Ballers had different plans: they won both the first half and the second half, becoming the winningest team in Pioneer League history.

All of this winning meant that the Ballers secured their postseason fate remarkably early in the season, which presented us at Oxide with a wild opportunity: as we were planning our annual internal meetup (which we call OxCon), we were realizing that the last night of the meetup was going to fall on the first postseason game at home. We felt we had to take the Oxide team to see the Ballers, and it’s a credit to our idiosyncratic team at Oxide that they embraced the idiosyncratic one at Raimondi. My colleague iliana — a Mariners fan! — felt that we should explain baseball to our fellow nerds, and the final session of OxCon was the two of us explaining what people were about to see. (Or tried to, anyway — baseball defies quick explanation!) We piled into busses and headed over to Raimondi, with kids and partners who were local meeting us at the game. We knew that we had kept our Oxide colleagues (very) busy over the week of OxCon, and told them that they should buy Ballers gear on Oxide’s nickel for anyone back home who might appreciate it. The game was awesome: the Ballers won — and Oxide employees went home loaded with swag for their families.

For the Ballers, their season continued. The three-game series went to the win-or-go-home third game, and after a gripping 1-0 victory, the Ballers advanced to the championship against the Idaho Falls Chukars. The five game set had a familiar format: two games away followed by three at home. The Ballers lost their first two games in Idaho Falls — badly. With the championship and the season on the line, the team returned to Oakland, where they needed to win three straight, starting on Friday night. My daughter was stoked: "Can you imagine the atmosphere on Sunday?" she excitedly asked. I pointed out that the team was going to be lucky to get the chance to play Saturday, let alone Sunday.

But my daughter’s optimisim proved to be prophetic. The team won decisively on Friday night — and again on Saturday night. And the atmosphere was indeed singular on the sold-out Sunday night. The Ballers got a good start with an early 3-run homerun — and then blew the game open with another 3-run jack to deep center in the 8th. Going into the top of the 9th inning, the Ballers were up by 8 runs. Any baseball fan will tell you that one bad inning can sink a game, that there is no lead large enough to assure a victory. Still, this felt pretty good.

The boys are away at college now, but they were with us in spirit, and my 18-year-old was listening from college in Washington. With two outs, he texted me: "Film this last out." I filmed it: a swinging strikeout as the stadium erupted. Oakland Fire, which would often position a fire engine in the outfield and sit atop it to watch the game, turned on their lights in celebration. The team stormed out of the dugout and mobbed their pitcher. And then, magic: that fire engine in deep center pointed a stream of water skyward and onto the outfield. The players instinctively ran to the water, like kids running through a sprinkler. They slid in the outfield, making snow angels on the wet ground. It was everything great about sports, in a single joyous moment.

These players know the significance of it all: they — like Oakland — have something to prove. The Pioneer League is a development league, so while we see some returning players next year, many will also move on. Some will move to other independent leagues, chasing their major league dream. Others will presumably hang it up, just as everyone hangs it up eventually. But for all of them, wherever their careers may take them, they will forever remember that electric Sunday night in West Oakland, when they proved what is possible when we refuse to give up on ourselves.

At long last, there’s a parade in Oakland.

Systems Software in the Large

Thu, 18 Sep 2025 00:00:00 +0000

Software is hard (yes, even in an era of vibe coding), and systems software — the silent engine room of modern infrastructure — is especially so. By design, systems software provides an abstraction for programs, insulating programmers from the filthy details that lie beneath; piercing that abstraction to implement the underlying system is to embrace those details and their gnarly implications. Moreover, the expectation for systems software is (rightly) perfection; a system that is merely functional can be deceptively distant from the robustness required of foundational software. Systems software isn’t the only kind of hard software, of course, and indeed software can be difficult just by nature of its scope and composition: it is hard to build software that is just… big. Software that consists of many different modules and components built by multiple people over an extended period of time is known as programming in the large, and its difficulties extend beyond the mere implementation challenges of systems software.

Because the difficulties with developing systems software are broadly orthogonal to those of programming in the large, intersecting these two challenges — that is, developing systems software in the large — is to take on the most grueling of projects: it is the stuff of which mythical man months are literally made. Why would anyone ever develop such a system? Because they are often necessary to tackle software’s equivalents of the wicked problem: problems that are not only never completely solved, but also not even really understood until implementation is well underway. There are not pat answers for developing these systems — nor, infamously, silver bullets — they’re just… brutal.

This is on my mind because of a talk that we had at OxCon last week. OxCon is our affectionate name for the annual Oxide meetup here in Emeryville, and it’s a highlight of the year for everyone at Oxide. This year more than lived up to our high expectations, replete with cameos from the extraordinary IBM 26 Interpreting Card Punch and the Oakland Ballers. At OxCon we like to both reflect back and look forward, so in that spirit, we asked Oxide engineer Dave Pacheco if he might be willing to present on the project he’s been leading the charge on for the past two years: software update.

When we shipped the first Oxide rack two years ago, it had the minimum functionality necessary to update all its software in the field. Our priority was to make this update mechanism robust over all else, and we succeeded in the sense that it is indeed robust — but the experience is not yet the seamless, self-service facility that we have envisioned. Software update for the Oxide rack is exactly the kind of wicked problem that necessitates systems software in the large: it is not merely dynamically overhauling a distributed system, but doing so while remaining operable in the liminal state between the old software and the new. Compounding this was the urgency we felt: delivering self-service update is essential to realize our vision of the cloud experience on premises, and our customers needed it as soon as we could deliver it. As if this weren’t enough, the Oxide update problem has an acute constraint not faced by the public clouds: we need to be able to deliver updates across an air gap — we cannot rely on the public cloud’s hidden crutch of operators and runbooks. It is a problem so wicked, you can practically hear it cackle.

Despite the thorniness of the problem, Dave and team had managed to achieve the ambitious milestones that they had set for themselves at OxCon last year, and I was naturally excited for his presentation this year. That said, I wasn’t ready for what was coming: Dave not only described the tremendous work on software update (delving into both the multi-year history of the project and the significant progress since the last OxCon), but also reflected on leading the software update project itself. The result was an absolutely extraordinary talk, not just on the mechanics of software essential to Oxide, but on the unique challenges of systems software in the large.

Dave’s talk dripped with hard-won wisdom, running the gamut from maintaining focus (and the looming specter of what Dave calls "organizational procrastination") to fighting scope creep and the mechanics of specific technical decisions. We felt Dave’s talk to be too good to be kept to ourselves — and thanks to our transparency, nothing in it needs to be secret; we are thrilled to be able to make it generally available:

This talk is a must watch for anyone doing systems software in the large, containing within it the kind of lessons that are often only learned the hard way. While we think it’s valuable for everyone, should you be the kind of sicko inexplicably drawn to exactly the kind of nasty problems that Dave describes, consider joining us — there is more systems software in the large to be done at Oxide!

Our $100M Series B

Wed, 30 Jul 2025 00:00:00 +0000

Note: Thie was co-authored with Steve Tuck, and originally appeared on the Oxide blog.

We don’t want to bury the lede: we have raised a $100M Series B, led by a new strategic partner in USIT with participation from all existing Oxide investors. To put that number in perspective: over the nearly six year lifetime of the company, we have raised $89M; our $100M Series B more than doubles our total capital raised to date — and positions us to make Oxide the generational company that we have always aspired it to be.

If this aspiration seems heady now, it seemed absolutely outlandish when we were first raising venture capital in 2019. Our thesis was that cloud computing was the future of all computing; that running on-premises would remain (or become!) strategically important for many; that the entire stack — hardware and software — needed to be rethought from first principles to serve this market; and that a large, durable, public company could be built by whomever pulled it off.

This scope wasn’t immediately clear to all potential investors, some of whom seemed to latch on to one aspect or another without understanding the whole. Their objections were revealing: "We know you can build this," began more than one venture capitalist (at which we bit our tongue; were we not properly explaining what we intended to build?!), "but we don’t think that there is a market."

Entrepreneurs must become accustomed to rejection, but this flavor was particularly frustrating because it was exactly backwards: we felt that there was in fact substantial technical risk in the enormity of the task we put before ourselves — but we also knew that if we could build it (a huge if!) there was a huge market, desperate for cloud computing on-premises.

Fortunately, in Eclipse Ventures we found investors who saw what we saw: that the most important products come when we co-design hardware and software together, and that the on-premises market was sick of being told that they either don’t exist or that they don’t deserve modernity. These bold investors — like the customers we sought to serve — had been waiting for this company to come along; we raised seed capital, and started building.

And build it we did, making good on our initial technical vision:

We did our own board designs, allowing for essential system foundation like a true hardware root-of-trust and end-to-end power observability.
We did our own microcontroller operating system, and used it to replace the traditional BMC.
We did our own platform enablement software, eliminating the traditional UEFI BIOS and its accompanying flotilla of vulnerabilities.
We did our own host hypervisor, assuring an integrated and seamless user experience — and eliminating the need for a third-party hypervisor and its concomitant rapacious software licensing.
We did our own switch — and our own switch runtime — eliminating entire universes of integration complexity and operational nightmares.
We did our own integrated storage service, allowing the rack-scale system to have reliable, available, durable, elastic instance storage without necessitating a dependency on a third party.
We did our own control plane, a sophisticated distributed system building on the foundation of our hardware and software components to deliver the API-driven services that modernity demands: elastic compute, virtual networking, and virtual storage.

While these technological components are each very important (and each is in service to specific customer problems when deploying infrastructure on-premises), the objective is the product, not its parts. The journey to a product was long, but we ticked off the milestones. We got the boards brought up. We got the switch transiting packets. We got the control plane working. We got the rack manufactured. We passed FCC compliance.

And finally, two years ago, we shipped our first system!

Shortly thereafter, more milestones of the variety you can only get after shipping: our first update of the software in the field; our first update-delivered performance improvements; our first customer-requested features added as part of an update.

Later that year, we hit general commercial availability, and things started accelerating. We had more customers — and our first multi-rack customer. We had customers go on the record about why they had selected Oxide — and customers describing the wins that they had seen deploying Oxide.

Customers started landing faster now: enterprise sales cycles are infamously long, but we were finding that we were going from first conversations to a delivered product surprisingly quickly. The quickening pace always seemed to be due in some way to our transparency: new customers were listeners to our podcast, or they had read our RFDs, or they had perused our documentation, or they had looked at the source code itself.

With growing customer enthusiasm, we were increasingly getting questions about what it would look like to buy a large number of Oxide racks. Could we manufacture them? Could we support them? Could we make them easy to operate together?

Into this excitement, a new potential investor, USIT, got to know us. They asked terrific questions, and we found a shared disposition towards building lasting value and doing it the right way. We learned more about them, too, and especially USIT’s founder, Thomas Tull. The more we each learned about the other, the more there was to like. And importantly, USIT had the vision for us that we had for ourselves: that there was a big, important market here — and that it was uniquely served by Oxide.

We are elated to announce this new, exciting phase of the company. It’s not necessarily in our nature to celebrate fundraising, but this is a big milestone, because it will allow us to address our customers' most pressing questions around scale (manufacturing scale, system scale, operations scale) and roadmap scope. We have always believed in our mission, but this raise gives us a new sense of confidence when we say it: we’re going to kick butt, have fun, not cheat (of course!), love our customers — and change computing forever.

The end game for John Fisher

Sun, 29 Jun 2025 00:00:00 +0000

A year and a half ago, I wrote about the unconscionable John Fisher, the inept, soulless owner of the formerly-Oakland A’s. When the team announced that it would be temporarily crashing on Sacramento’s couch, I pointed out that the incompetence of this ownership knows no depth or bounds: they they will always find new ways to screw up.

Now we are near the mid-point of that first season in Sacramento, and Fisher hasn’t disappointed. In fact, the creativity of his innate incompetence continues to surprise, as the team has refused to even adopt the name of their host city: they are not called the "Sacramento A’s", but rather just "The Athletics" — devoid of a city name entirely. Sacramento is a baseball town, but it’s also a proud spot with a rightful chip on its shoulder, overlooked among California’s larger cities despite its obvious importance to the state; Fisher refusing to so much as put the city’s name on the team’s chest is a grievous misread of the town’s character. And it gets worse: because the Las Vegas Convention and Visitors Authority has formally sponsored the team, fans in not-to-be-uttered Sacramento must endure the further indignity of "Visit Las Vegas" signs plastered all over their own outfield. Add to this the highest ticket prices in MLB, and in just a few months John Fisher has achieved in Sacramento what took him years in Oakland: fans that hate his guts. And not only is attendance predictably bad in Sacramento, the on-field product is also affected, with ace Luis Severino openly talking about how much he dislikes the minor league park.

Okay, so Sacramento is a listing wreck; how are things coming along with the team’s new home in Las Vegas? If anything, it’s worse — and there are so many problems with the Vegas stadium as conceived, it’s hard to know where to start. The small site and scorching summer temperatures have necessitated a fixed dome, which is infamously problematic in baseball; the on-strip location is terrible for locals; the attendance projections are delusional; and the planned on-site Bally’s hotel/casino is unlikely to ever be built.

And these were the problems that were clear two years ago; since then, everything has more or less moved against Fisher’s plans in Vegas. First, on the demand side, Vegas has hit economic headwinds, for myriad reasons: Vegas has gotten much more expensive; gambling is not exclusive to Vegas as it once was; international tourism is way down. Any decline to tourism doubly hits Fisher: not only is he relying on tourists for a laughable 8,000 fans per game, but with the Vegas economy itself so dependent on tourism, any decline will surely affect the local entertainment dollar as well.

On the supply side, it’s even grimmer: steel tariffs have surged, and construction costs are up across the board. And the tight schedule makes it more expensive: by not having yet started construction but with an immovable date of Opening Day 2028, stadium construction will end up costing much more. So the stadium that was once going to cost $1.5B is now north of $1.75B — and on its way to $2B.

I bring all of this up now because this past Monday was a "groundbreaking" for the stadium — which earned its air quotes when the needlessly hardhatted dignitaries drove their shovels not into the actual ground, but rather an indoor planter (?!). This is already pretty bad, but groundbreakings are after all PR events, and performative digging is to be expected. However, if there’s one dimension you can count on John Fisher to be truly groundbreaking, it’s in new levels of incompetence, and he delivered on Monday: as Doug Puppel of The Engineering News-Record reported, the construction equipment in the background was rented as props for the day. Yes, in this Potemkin groundbreaking, even the construction equipment was a sham. Why not use, like, actual construction equipment? Because there isn’t any — and there isn’t any, because construction hasn’t actually started. And construction hasn’t started because John Fisher doesn’t have the money.

But wait, isn’t he a billionaire? Yes, but across essentially three illiquid assets: his Gap inheritance; the San Jose Earthquakes; and the A’s. In terms of money towards the project, he has the (infamous) $350M from the state of Nevada, and then $300M in debt from Goldman Sachs. The only new source of funding in the past year is a $175M deal with Aramark — though as David Samson points out, a deal with a concessionaire as a part of stadium construction isn’t that notable. (Also, aside: Aramark sucks.)

All of this leaves John Fisher needing (at least) ~$1B. It’s easy to get numb to dollar figures, so take it from someone who has raised a bunch of venture capital: that is a lot of money. As I discussed over a year ago, he can find capital in one of three ways: he can take on debt, he can sell equity, or he can pony up.

Debt is the instrument one would normally go to for a large construction project, as it’s non-dilutive financing. But as Todd Saran’s thread from 2024 elucidates, those who underwrite debt — banks — are going to look at cash flow: they need to know that they can get paid back! Debt analysts are going to care quite a bit about (say) the A’s fanciful attendance projections. Traditional lenders will find plenty to puke on in the A’s plans (and the $300M in debt from Goldman assuredly has some gnarly conditions that assures that they will get theirs). Now, there may be some other pockets of debt out there. Over at The Rickey Report, Alex Espinoza looked into some MLB sources of funds, but these too are finite. Many people believe that "MLB won’t let Vegas fail", but the numbers in Vegas are too large for MLB to bail Fisher out alone — and the two facilities that seem to be available from MLB to extend debt to a single team amount to just ~$225M.

If debt options are exhausted, the other possibility — and the one that John Fisher has been chasing for the last two years — is to sell equity. This is what a venture-backed business does to raise money: it sells off a piece of itself, using the proceeds to run the business. However, there is a basic problem with selling equity in the A’s to fund the stadium in Vegas: someone needs to buy it. And someone buying equity needs to make money, which they will do by selling their equity later at a higher price. So when you’re buying equity in a private company, there is the obvious question: what is the price, anyway? (That is, what is the company worth?) And what will it be worth in the future? (What is the company’s growth in valuation?) The kinds of questions that equity investors ask are different from banks: namely, not how can you pay me back but how are you growing? Problematically for anyone looking at buying equity, the A’s aren’t growing, they’re shrinking: they have not only nuked their entire fanbase (and clearly not won any new fans in Sacramento!), they have had replies off on all of their social media accounts since 2023. It is hard to imagine a bigger red flag for a consumer-facing brand: a brand that has so thoroughly alienated so many customers that it lives in fear of customers entirely.

That said, for a sufficiently low price, equity investors could be enticed, and this is where it gets sticky for Fisher: despite the encouragement of A’s fans, he does not, in fact, want to give up a controlling interest in the team. For the math to work, the valuation needs to be eye-wateringly high: Forbes pegged the value of the A’s at $1.2B (when they had a stadium!) and the Orioles sold for $1.725B last year, but neither of those valuations is really high enough for Fisher to raise the kind of dough he needs without giving up control. According to CNBC reporting, the A’s were looking to raise $550M at a $2B enterprise valuation which is both absolute lunacy and exactly what Todd Saran predicted eighteen months ago. This price is delusional because it’s pricing in a bunch of things that haven’t happened yet (like building the stadium!), and the A’s will be on the hook for any cost overruns — and if they default on their debt in the process, the equity will be worth zero. (If you are, like me, a David Samson fan, you hear that "zero" followed with rhetorical florish like "zilch!" and "nada!") Added to all of the other problems in this deal: it’s been out there for a long, long time. If you are the kind of person or institution who is a candidate to participate in this deal, you have already seen it — and passed on it.

If debt and equity are both off the table, this leaves the last option, and to those with a cursory view of the A’s, it’s the easiest and most obvious: this dude is a billionaire — can’t he just… write a check?! And here we get to the illiquidity of those other assets: his Gap inheritance and the San Jose Earthquakes. There are lots of reasons why Fisher would rather not part with either: not only does it run counter to his base instinct of hoarding his wealth, but he will also owe significant federal and (presuming he still lives in San Francisco) California state taxes. But it’s an indicator of just how exhausted the other paths are that this does indeed seem to be the path that Fisher is taking: it was announced earlier this month that he is looking to sell a controlling interest in the San Jose Earthquakes. This isn’t the only MLS team up for sale (the Vancouver Whitecaps are also on the block), and it’s not clear how much he’s selling (just that it’s a controlling interest) or at what price, but the team is worth $600M — there’s surely money to be raised by selling the stake. There’s the Gap, too: at Friday’s close ($21.90), Fisher’s 45.8M shares are worth ~$962M. All of this wealth is very significant, but the assets are not enough to get Fisher all the way there: even if he were to unload the totality of these holdings (at, say, $1.5B), he would owe so much in tax (up to ~37%!) that it would significantly reduce the capital available to the stadium. And he would still be on the hook for any construction overage — and without any financial backstop whatsoever.

So Monday was significant in that it has broken ground on something very important: the end game for John Fisher. I don’t think Fisher will find a substantial investor (it’s hard to find the dumb money when you are, in fact, the dumb money), but it’s not inconceivable that he will liquidate a significant amount of his own assets to pay for the stadium and its inevitable overages. I view this as unlikely, but I would honestly be delighted by it: not only for the fortune he would owe in taxes, but also because he would be staking his own inherited wealth on his desert folly — and there would be a very real possibility that he would lose it all. It would be gloriously fitting for John Fisher, whose two existential fears are clearly that he is an incapable dunce and that he will fumble his birthright, to manage to confirm both in a single, spectacular disaster.

But the far more likely outcome is that Fisher fiddles while Sacramento and Las Vegas burn. In Sacramento, season ticket holders — having been badly burned by the John Fisher experience — will not re-up. The MLB players union, long underserving its own membership by allowing this travesty, will demand significant changes for next year. Fisher himself will hold out for an equity investor in the A’s, finding a way to botch (or delay) a sale of the Quakes. In Vegas, the progress on the stadium will be essentially nil, awaiting not merely the financing but the many missing pieces needed to begin construction. As time passes on the quiet site, it will be obvious that they are stalled, even by the comically vague schedule standard that the A’s have set for themselves.

Finally, at some point in the fall or perhaps early winter, it will all snap: it will be incontrovertable that there will be no opening day in Vegas in 2028. As he has done so many times before, Fisher will first try to imply that a later timeline was the plan all along — but with Sacramento unsustainably molten, I suspect that MLB’s patience may finally be exhausted. If past is any prediction, Fisher will blame both Sacramento and Las Vegas (or perhaps an Oakland conspiracy?), decry how hard all of this has been on him and his family, and begin talk of a new location — or (more likely) a new city.

On the one hand, this will be the moment that Salt Lake City has been waiting for, but on the other, I suspect that the chorus of warnings from Oakland about John Fisher will carry to Utah — especially when joined by new voices from Sacramento and Las Vegas. MLB, at long last, will force the sale. This is still in the future, of course, but it’s now foreseeable: what John Fisher broke ground on on Monday is — I believe — the end game of his ownership. I just want to be there when the new ownership turns the replies back on!

College Baseball, Venture Capital, and the Long Maybe

Sun, 15 Jun 2025 00:00:00 +0000

My son is a college baseball player. One of the singular aspects of having kids is that they open your eyes to facets of humanity that you might not have seen otherwise. I was not a college athlete (and knew little of even baseball until he got the bug at age seven), so I have learned much along the way. You — perhaps like me several years ago — may have an idea of how college athletics proceeds: that a kid plays well in high school; that it informs the schools to which they apply; that they have some conversations with coaches when considering where to matriculate; that they go to a school; that they play for four years.

But this is naive to the point of fantasia.

In fact, college athletics is a parallel universe that may share some of the same classes and physical plant as a non-athletic experience, but is wholly different in who matriculates and how: in order to play on an NCAA varsity team, one has necessarily dedicated their young lives to their sport — and the sport is almost assuredly the lens through which school decisions have been made. (As an aside, this is what makes the Varsity Blues scandal ever the more revolting: not only was it crooked and fraudulent, but it made a mockery of the athletes that worked so hard to earn a spot that was in fact sold to a non-athlete.)

This parallel universe has been true for all NCAA sports for a long time, but for the revenue sports (football, basketball, hockey, baseball), the higher stakes for both the institution (for whom these sports represent revenue) and for the athlete (who dreams for a shot to play professionally) have made for an even more tumultuous recruiting process. To this dry tinder, the spark of Name, Image, and Likeness (NIL) money and the explosion of the NCAA transfer portal lit a conflagration that has permanently altered college sports — and the path of the student-athlete today looks simply nothing like their non-athlete peers.

So what does it look like? Early on in my own son’s journey — as he was navigating different junior college options in the spring of 2022 — it struck me that the experience of the college athlete does have a clear analogue, and it is in fact one in which I do have recent and germane experience: becoming a college athlete (especially for revenue sports at the highest levels of play) looks uncannily like raising a round of venture capital.

Raising venture capital

Raising venture capital, it should be said, has been an absolutely wild experience that (for me) had no real life precedent: an emotional roller coaster in which highest-highs and lowest-lows seem to happen simultaneously. The further my son went into his baseball career — and especially as he finished up at JUCO and went through the process of finding an NCAA home at which to play — the clearer this analogy became to me.

I want to explore the parallels in some depth. While I suspect that there is some applicability here to all NCAA varsity sports, I’ll stick to NCAA baseball. If you have raised venture capital but don’t know the first thing about college baseball, you’ll find that you’re a very quick study; if you know a bunch about college baseball and don’t know anything about venture capital, you might be interested to know that your experience has a clear analogue in another domain. (And if you know neither of these things, I am going to try to introduce you to both a bit!)

Why the parallels?

Raising a round of venture capital and landing a college roster spot are, in the abstract, both a kind of asymmetric, non-linear coupling. In both cases, you have an institution that is placing a bet on the future. (And the future is — as baseball player Yogi Berra famously quipped — especially tough to make predictions about!) The bet, once placed, can’t be taken back: once the money is wired or the roster slot is given, a die is cast. The decision isn’t forever, of course (you can choose not fund subsequent rounds and you can make future roster adjustments) but in both cases there is an opportunity cost associated with the decision that is irreversible.

And the stakes are high. In both cases, the institution is results-oriented: venture capital firms need to deliver for their limited partners and have a constant eye on raising their next fund; coaches need to win and constantly fear for their job security. For the counterparty, the stakes are differently high: the athlete — like the entrepreneur — is trying to manage their own fate rather than a portfolio; they don’t get to apply portfolio theory to the one career they get to live.

The relationship with risk is also tricky with both. On the one hand, the institutions can use portfolio theory to mitigate their downside risk (a company that has no return for investors, a player that doesn’t work out), but the greater risk is arguably passing on someone that would have been a difference maker — the fear of missing out (FOMO).

The parallels

Here are a few VC’isms — and how they map to the experience of college baseball players:

The Pitch Deck. The deck is often the introduction of a venture capitalist to a company, sent after a warm intro. For the athlete, the pitch deck consists of video and some metrics, posted on Twitter (the LinkedIn of college baseball). In both cases, the best story is (naturally) being told: they aren’t dishonest, but they also aren’t going to emphasize flaws or risks. In my experience, pitch decks are important, but not as important as the underlying business; similarly, terrific video of a bullpen or batting practice can’t turn a baseball player into someone they aren’t.
The Long Maybe. VCs need to invest out of a fund (that is, they need to put their capital to work), and FOMO leaves them nervous about turning down companies prematurely — they become reluctant to tell a company "no", even if it’s plain that they will never summon the conviction to invest. VCs call this "preserving optionality", but it is in fact a "long maybe" — and it leads to long paths-to-nowhere with VCs that seemingly always want something else to prolong the conversation, with desultory requests for a new conversation about how the recent quarter went, or perhaps an introduction to just one more customer. (Often with long periods of ghosting in between!)

Just as VCs need to invest, coaches need to fill a roster, and baseball players will immediately recognize the coaches that are in and out of their DMs — with slow-moving, swirling processes. Often, coaches will blame external events for their inaction ("we need to wait until the House ruling", "we need to wait until the portal opens", etc.); VCs, too, will blame other events ("we have been busy raising a new fund", "we’ve been in Europe the past month"). The truth in both cases is that the institution just isn’t that interested; both startup and athlete would be well-advised to pencil them in as a "no" and move on.
The Term Sheet. When a VC chooses to lead a round, they will write a term sheet that contains their desired structure. Getting to a term sheet is a huge milestone, but the round isn’t closed yet: it needs to be negotiated (and signed!), long-form docs needs to be drafted, diligence needs to be done, and then the investment wired to actually close the round. (To say nothing of follow-on capital that often needs to be raised!) Term sheets are (broadly) non-binding, and many things can go wrong between a term sheet and a round closing.

Offers in college baseball are similarly complicated. What feels like an offer may evaporate — and even what feels like the surely binding elements of an offer (e.g., a roster spot?!) are not in fact binding at all. So how does one know the veracity of an offer? As with VCs, one is ultimately dependent upon the integrity of a coach. Fortunately, most coaches — like most VCs! — have high integrity. But it is not all of them, and there certainly exist unscrupulous coaches (and VCs!) who are able to hide their misbehavior, dependent on the fact that there is no transparency in their actions. These coaches routinely overrecruit (i.e., offer commitments for more roster slots than they in fact have) or otherwise rescind offers. These coaches become infamous to players over their careers, and the athletes that they have wronged live for the opportunity to face these coaches and win. (Having seen a dramatic walk-off under exactly these conditions, I can tell you that it was electric to watch mass vindication, a literal dogpile of spurned players who had just proved they were the better team.)
The Preempt. An outgrowth of FOMO and preserving optionality are VC firms that want to "preempt" a fundraise — to write a term sheet for a company that isn’t raising. On the one hand, this seems great: it’s really distracting to raise, and if a particularly interested firm wants to frontrun everyone by getting ahead of your next round, why not let them? And while it certainly can be great, it can also be a hot mess: if the preempting firm drags their feet, they may end up distorting the process that a company would have run otherwise — which can be calamitous if the preemption deal falls apart after having implicitly denied the company of runway by wasting their time.

For high school ballplayers in particular, the preemption takes the form of coaches making "verbal commitments" to high school students who are not seniors. These really shouldn’t have "commitment" in the name as they are entirely non-binding; coaches walk away from these so-called commitments all the time. Meanwhile, the kid (and they are kids — tautologically under 18!), believing that they have found a home, has stopped their own recruiting process as a junior (or even a sophomore!), depriving themselves of getting the best possible post-high school opportunity.

While this didn’t (exactly) happen to my son, I have seen it happen too many times to not channel my inner parent: the NCAA needs to forbid and punish this diabolical practice. (As for VCs that preempt and then walk away from a signed term sheet, the NCAA may not be coming for you — but karma will.)
The Exploding Offer. When a VC firm writes a startup a term sheet, it wants it signed as soon as possible. Term sheets therefore often have exploding terms where they are only valid for a short period of time. Offers for ballplayers can be exploding too, and in fact it might be more common: teams have a finite amount of time to put their roster together. In both cases, this is a term that can be easily negotiated away — and it can become a red flag if (say) a VC won’t allow you to reference check their portfolio or a coach won’t allow you to visit a campus (both emanately reasonable justifications to extend the deadline on an offer).
Multiple Term Sheets. The fantasy for every startup is multiple term sheets, as it represents leverage to get terms adjusted or to otherwise get the best possible deal. For startups, this certainly happens, but it’s not as common as one might think: getting a term sheet from one VC with the intent of forcing the hand of others may in fact just get others to decide that they can’t get there. (Or worse: slowly decide that they can’t get there!) For college baseball players, this feels more common, but still may be broadly rare. In both cases: it’s a great problem to have, and it’s worth really sitting down to think about the rubric for making a decision. Why would you not just take the economically better deal always? An excellent segue…
The Valuation Overhang. When you raise a round of venture capital, you do so at a valuation. For founders, a mistake is optimizing exclusively for the best deal economics, which can result in either raising too much money or raising money at too high a valuation. On the one hand, this is a good problem to have: if the valuation for a startup is being driven up, it may indicate a frothy market that an entrepreneur wants to take advantage of. On the other, though, raising at too high a valuation is perilous: the high valuation can create expectations that the company can’t possibly live up to — and the valuation itself serves to deprive a company of options. (As it has in so many other ways, HBO’s Silicon Valley absolutely nails the peril of raising too much.)

For college baseball, the valuation overhang would be going to a Power Four conference school straight out of high school. Some extraordinary high schoolers can compete at that highest of levels, but for others, it’s just too big a step: they end up redshirting and then with limited playing time (or none) their second year, they realize that they aren’t going to play — and they enter the transfer portal. Despite being a standout high school athlete, these players can find that their limited college careers may result in them being perceived as much riskier than a known JUCO or NAIA player.
The Down Round. In venture capital, raising at a valuation less than your previous valuation is called a down round. In college baseball, the down round analogy is entering the transfer portal with a destination that is a lower division. (To give perspective on the madness that is the transfer portal, as I write this there are 5,700 baseball players in the transfer portal — and there are ~10,000 total Division 1 baseball players.) While a down round is undesirable for many reasons, it means that a company is at least finding a path to survival; for a player going backwards in the portal, they are trying to find a path to play — and that year in JUCO as a bounceback may be exactly what they need to rebuild and return to the highest levels of play.
The IPO. The dream for every startup is an initial public offering — and the dream for every ballplayer is to play Major League Baseball. At some level, both dreams are ludicrous, but they also surely seemed outlandish many times over to the people who realized them. The romance of both VC and college baseball is that these things can and do happen!

Navigating it all

To the degree that the analogy is helpful or instructive, it may be to anyone engaged in either of these two insane processes — that there might be something to learn from the other. My advice to entrepreneurs and athletes alike:

Figure out what you want. For most startups, it’s to be a big, successful, public company; for most college baseball players, it’s to be the best player they can possibly be — and to get a shot at getting paid to do it. But this isn’t everyone, and if your goals are different, you should figure out what they in fact are!
Keep your goals in mind when making decisions. If your goal is to make the company successful, it may drive you to an investor or a structure that isn’t obvious; if your goal is to improve as a player, it may drive you to a program or a coach that others have overlooked. Know your own goals, and know that you don’t owe anyone an explanation for your decision.
Go where you’re wanted. It feels tautological that anyone getting across the line with an offer wants you — but not all offers are the same. Be wary of where you are not really wanted, and note that conviction and praise are not the same thing: anyone wanting to get you over the line will tell you what you want to hear. Find the real conviction, and you will find an institution that is with you not just on sunny days but stormy ones too.

Finally, just because I’m penning this on Father’s Day, let me add a personal note. To all of the college baseball players out there, from the JUCO bandits and the D1 bouncebacks to the mid-major grinders and the Power 4 phenoms: you are getting an education far beyond the classroom. Sometimes you have had to endure bad behavior by adults that others won’t have to suffer until they are much older — and you certainly face a level of pressure that most of your collegiate peers don’t and won’t know. We, your parents, see it, and we are proud as hell of your grit and resolve; go get 'em, kid!

RIP USENIX ATC

Sun, 11 May 2025 00:00:00 +0000

USENIX made the decision this week to discontinue its flagship Annual Technical Conference. When USENIX was started in 1975 — before the Internet, really — conferences were the fastest vector for practitioners to formally share their ideas, and USENIX ATC flourished. Speaking for myself, I came up lionizing ATC: I was an undergraduate in the early 1990s, and programs like the USENIX Summer 1994 conference felt like Renaissance-era Florence for systems practitioners.

When we developed DTrace in the early 2000s, we knew that there was no better venue to announce it to the world than USENIX ATC. We put a lot of effort into the resulting paper, Dynamic Instrumentation of Production Systems — and I was elated when it was accepted for USENIX ATC 2004.

The conference itself was… surprising. This was not the USENIX of a decade prior; the conference was decidedly academic, and in the strictest sense: all of the presentations were from PhD students seeking out academic work. I wrote about this (with characteristic bluntness?) after I returned from the conference, asking whither USENIX? This blog post engendered quite a bit of discussion (the bygone era of meaningful discourse in blog comments!) — and an especially thoughtful long-form response from Werner Vogels.

Werner’s thoughts in turn inspired my own response, where I in particular looked at the Program Committee formulation — and became alarmed by the total collapse in industrial participation. Ted Leung noted this, and referred us to Rob Pike’s infamous polemic. I wrote again on this later that summer (those halcyon days immediately before the delightful distraction of a newborn!), asking whither systems research?

The summer of 2004 was a long time ago; the occasion of ATC’s discontinuance affords us the opportunity to look back on all of this now two decades on.

First, on systems research, it is ironic (or perhaps it isn’t?) that Pike decried the lack of new systems — but one of his own creation, the Go programming language — became indisputably one of the most important system software developments of the 2010s. Go is not alone, of course: that decade also brought us the Rust programming language, a language that I fell so in love with that we named our company after it.

What happened here? As in so many other dimensions, I think our discussion and consternation in 2004 plainly underestimated the importance and impact of open source. Go and Rust do not arise — cannot arise — in a proprietary world. These are also not the work of hobbyists: they were written by professionals for professionals in a professional capacity. And of course, it isn’t just languages: in all manners of systems software, the leading edge of innovation is in deployed, production, open source systems. Open source systems are not without their own complexity and conflict, of course — viz. the recent shoot-out at the CNCF corral — but these are literal sideshows: in the last two decades new, innovative, production-oriented systems have thrived (whether we call those systems "research" or not). Importantly, the vector for publishing for the practitioner is via repositories rather than being confined to only formal writing.

Second, as for USENIX ATC itself, the conference itself plainly struggled. The problems that I had seen with respect to an overly academic conference seemingly metastasized, and for me, the conference drifted further and further from the shores of practitioner relevance. (For practitioners seeking to formally publish, I recommended that they instead target Communications of the ACM via its practitioner-focused content.)

Seemingly unaware of my own ambivalence about the conference, however, USENIX asked me to return in 2016 to give the ATC keynote. With the caution that some found my presentation offensive, my keynote, A Wardrobe for the Emperor, outlined my belief that computer science had erred grievously in insisting upon conferences (rather than journals) as a publishing vehicle — and that USENIX and its conferences had been a casualty. And while my keynote may have been particularly candid, I am not the only one who saw this: Rik Farrow wrote up an excellent piece on my talk for ;login: giving context that I was unaware of — namely that inside of USENIX, this was an issue of concern reaching back decades. So USENIX ATC did finally succumb, and I do view it as a casualty here, asphyxiated by the rough love of academic computer science and its inability to grow beyond the conference model of publishing.

That said, while ATC may have died in the arms of academia, it would be unfair to say that academia alone killed it: as Adam and I discussed with Stephen O’Grady, KellyAnn Fitzpatrick and Theo Schlossnagle, conferences are hard — and in-person conferences are especially so. While they certainly lose something in the process, it feels like online conferences give us indisputably (overwhelmingly!) more bang for the buck; it felt like some decline of USENIX’s in-person conference model was inevitable. This is not to say that in-person models are impossible, of course, just that they needed to look different. (Selfishly, I wish USENIX had developed conferences like the Systems We Love event that we ran in 2016 — selfish because I’d much rather attend than organize!)

On ATC, I think we can fairly grieve what we lost: ATC was — at its height — a singular forum for presenting pioneering systems work. And even as it declined in attendance, it retained this spirit in its best works. Certainly, I view our own work at ATC in 2004 as striving to be in this tradition — and I think that it’s apt and fitting that last year’s Best Paper (which will stand as the penultimate USENIX ATC Best Paper) is on a system to validate AI infrastructure at scale, deployed on a real, production system, reporting results that are relevant to the practitioner.

So RIP, USENIX ATC. I know we had a complicated relationship over the years, but it was a good run. Thank you to everyone at USENIX who gave us an important forum for some of our biggest and boldest ideas — and we’ll always have the summer of 1994.

Oxide’s compensation model: How is it going?

Thu, 01 May 2025 00:00:00 +0000

How it started

Four years ago, we were struggling to hire. Our team was small (~23 employees), and we knew that we needed many more people to execute on our audacious vision. While we had had success hiring in our personal networks, those networks now felt tapped; we needed to get further afield. As is our wont, we got together as a team and brainstormed: how could we get a bigger and broader applicant pool? One of our engineers, Sean, shared some personal experience: that Oxide’s principles and values were very personally important to him — but that when he explained them to people unfamiliar with the company, they were (understandably?) dismissed as corporate claptrap. Sean had found, however, that there was one surefire way to cut through the skepticism: to explain our approach to compensation. Maybe, Sean wondered, we should talk about it publicly?

"I could certainly write a blog entry explaining it," I offered. At this suggestion, the team practically lunged with enthusiasm: the reaction was so uniformly positive that I have to assume that everyone was sick of explaining this most idiosyncratic aspect of Oxide to friends and family. So what was the big deal about our compensation? Well, as a I wrote in the resulting piece, Compensation as a Reflection of Values, our compensation is not merely transparent, but uniform. The piece — unsurprisingly, given the evergreen hot topic that is compensation — got a ton of attention. While some of that attention was negative (despite the piece trying to frontrun every HN hater!), much of it was positive — and everyone seemed to be at least intrigued.

And in terms of its initial purpose, the piece succeeded beyond our wildest imagination: it brought a surge of new folks interested in the company. Best of all, the people new to Oxide were interested for all of the right reasons: not the compensation per se, but for the values that the compensation represents. The deeper they dug, the more they found to like — and many who learned about Oxide for the first time through that blog entry we now count as long-time, cherished colleagues.

That blog entry was a long time ago now, and today we have ~75 employees (and a shipping product!); how is our compensation model working out for us?

How it’s going

Before we get into our deeper findings, two updates that are so important that we have updated the blog entry itself. First, the dollar figure itself continues to increase over time (as of this writing in 2025, $207,264); things definitely haven’t gotten (and aren’t getting!) any cheaper. And second, we did introduce variable compensation for some sales roles. Yes, those roles can make more than the rest of us — but they can also make less, too. And, importantly: if/when those folks are making more than the rest of us, it’s because they’re selling a lot — a result that can be celebrated by everyone!

Those critical updates out of the way, how is it working? There have been a lot of surprises along the way, mostly (all?) of the positive variety. A couple of things that we have learned:

People take their own performance really seriously. When some outsiders hear about our compensation model, they insist that it can’t possibly work because "everyone will slack off." I have come to find this concern to be more revealing of the person making the objection than of our model, as our experience has been in fact the opposite: in my one-on-one conversations with team members, a frequent subject of conversation is people who are concerned that they aren’t doing enough (or that they aren’t doing the right thing, or that their work is progressing slower than they would like). I find my job is often to help quiet this inner critic while at the same time stoking what I feel is a healthy urge: when one holds one’s colleagues in high regard, there is an especially strong desire to help contribute — to prove oneself worthy of a superlative team. Our model allows people to focus on their own contribution (whatever it might be).

People take hiring really seriously. When evaluating a peer (rather than a subordinate), one naturally has high expectations — and because (in the sense of our wages, anyway) everyone at Oxide is a peer, it shouldn’t be surprising that folks have very high expectations for potential future colleagues. And because the Oxide hiring process is writing intensive, it allows for candidates to be thoroughly reviewed by Oxide employees — who are tough graders! It is, bluntly, really hard to get a job at Oxide.

It allows us to internalize the importance of different roles. One of the more incredible (and disturbingly frequent) objections I have heard is: "But is that what you’ll pay support folks?" I continue to find this question offensive, but I no longer find it surprising: the specific dismissal of support roles reveals a widespread and corrosive devaluation of those closest to customers. My rejoinder is simple: think of the best support engineers you’ve worked with; what were they worth? Anyone who has shipped complex systems knows these extraordinary people — calm under fire, deeply technical, brilliantly resourceful, profoundly empathetic — are invaluable to the business. So what if you built a team entirely of folks like that? The response has usually been: well, sure, if you’re going to only hire those folks. Yeah, we are — and we have!

It allows for fearless versatility. A bit of a corollary to the above, but subtly different: even though we (certainly!) hire and select for certain roles, our uniform compensation means we can in fact think primarily in terms of people unconfined by those roles. That is, we can be very fluid about what we’re working on, without fear of how it will affect a perceived career trajectory. As a concrete example: we had a large customer that wanted to put in place a program for some of the additional work they wanted to see in the product. The complexity of their needs required dedicated program management resources that we couldn’t spare, and in another more static company we would have perhaps looked to hire. But in our case, two folks came together — CJ from operations, and Izzy from support — and did something together that was in some regards new to both of them (and was neither of their putative full-time jobs!) The result was indisputably successful: the customer loved the results, and two terrific people got a chance to work closely together without worrying about who was dotted-lined to whom.

It has allowed us to organizationally scale. Many organizations describe themselves as flat, and a reasonable rebuttal to this are the "shadow hierarchies" created by the tyranny of structurelessness. And indeed, if one were to read (say) Valve’s (in)famous handbook, the autonomy seems great — but the stack ranking decidedly less so, especially because the handbook is conspicuously silent on the subject of compensation. (Unsurprisingly, compensation was weaponized at Valve, which descended into toxic cliquishness.) While we believe that autonomy is important to do one’s best work, we also have a clear structure at Oxide in that Steve Tuck (Oxide co-founder and CEO) is in charge. He has to be: he is held accountable to our investors — and he must have the latitude to make decisions. Under Steve, it is true that we don’t have layers of middle management. Might we need some in the future? Perhaps, but what fraction of middle management in a company is dedicated to — at some level — determining who gets what in terms of compensation? What happens when you eliminate that burden completely?

It frees us to both lead and follow. We expect that every Oxide employee has the capacity to lead others — and we tap this capacity frequently. Of course, a company in which everyone is trying to direct all traffic all the time would be a madhouse, so we also very much rely on following one another too! Just as our compensation model allows us to internalize the values of different roles, it allows us to appreciate the value of both leading and following, and empowers us each with the judgement to know when to do which. This isn’t always easy or free of ambiguity, but this particular dimension of our versatility has been essential — and our compensation model serves to encourage it.

It causes us to hire carefully and deliberately. Of course, one should always hire carefully and deliberately, but this often isn’t the case — and many a startup has been ruined by reckless expansion of headcount. One of the roots of this can be found in a dirty open secret of Silicon Valley middle management: its ranks are taught to grade their career by the number of reports in their organization. Just as if you were to compensate software engineers based on the number of lines of code they wrote, this results in perverse incentives and predictable disasters — and any Silicon Valley vet will have plenty of horror stories of middle management jockeying for reqs or reorgs when they should have been focusing on product and customers. When you can eliminate middle management, you eliminate this incentive. We grow the team not because of someone’s animal urges to have the largest possible organization, but rather because we are at a point where adding people will allow us to better serve our market and customers.

It liberates feedback from compensation. Feedback is, of course, very important: we all want to know when and where we’re doing the right thing! And of course, we want to know too where there is opportunity for improvement. However, Silicon Valley has historically tied feedback so tightly to compensation that it has ceased to even pretend to be constructive: if it needs to be said, performance review processes aren’t, in fact, about improving the performance of the team, but rather quantifying and stack-ranking that performance for purposes of compensation. When compensation is moved aside, there is a kind of liberation for feedback itself: because feedback is now entirely earnest, it can be expressed and received thoughtfully.

It allows people to focus on doing the right thing. In a world of traditional, compensation-tied performance review, the organizational priority is around those things that affect compensation — even at the expense of activity that clearly benefits the company. This leads to all sorts of wild phenomena, and most technology workers will be able to tell stories of doing things that were clearly right for the company, but having to hide it from management that thought only narrowly in terms of their own stated KPIs and MBOs. By contrast, over and over (and over!) again, we have found that people do the right thing at Oxide — even if (especially if?) no one is looking. The beneficiary of that right thing? More often than not, it’s our customers, who have uniformly praised the team for going above and beyond.

It allows us to focus on the work that matters. Relatedly, when compensation is non-uniform, the process to figure out (and maintain) that non-uniformity is laborious. All of that work — of line workers assembling packets explaining themselves, of managers arming themselves with those packets to fight in the arena of organizational combat, and then of those same packets ultimately being regurgitated back onto something called a review — is work. Assuming such a process is executed perfectly (something which I suppose is possible in the abstract, even though I personally have never seen it), this is work that does not in fact advance the mission of the company. Not having variable compensation gives us all of that time and energy back to do the actual work — the stuff that matters.

It has stoked an extraordinary sense of teamwork. For me personally — and as I relayed on an episode of Software Misadventures — the highlights of my career have been being a part of an extraordinary team. The currency of a team is mutual trust, and while uniform compensation certainly isn’t the only way to achieve that trust, boy does it ever help! As Steve and I have told one another more times that we can count: we are so lucky to work on this team, with its extraordinary depth and breadth.

While our findings have been very positive, I would still reiterate what we said four years ago: we don’t know what the future holds, and it’s easier to make an unwavering commitment to the transparency rather than the uniformity. That said, the uniformity has had so many positive ramifications that the model feels more important than ever. We are beyond the point of this being a curiosity; it’s been essential for building a mission-focused team taking on a problem larger than ourselves. So it’s not a fit for everyone — but if you are seeking an extraordinary team solving hard problems in service to customers, consider Oxide!

Why Gelsinger was wrong for Intel

Sun, 08 Dec 2024 00:00:00 +0000

By all accounts, Pat Gelsinger is affable, technically sharp, hard-working, and decent. Those who have worked for him praise him as a singularly good manager. In January 2021, when Gelsinger was abruptly named the CEO of Intel, this is more or less all I knew of him — and I found myself urgently needing to learn much more.

To understand my urgency, it’s worth rewinding the clock back to late 2019 and the earliest days of Oxide. We knew what we wanted to build, but we had some big (consequential!) decisions to make — first among them our choice of host CPU. Based on what we had seen in AMD Naples and Rome, we felt that AMD was clearly outpacing Intel, but this was a big bet and we wanted to be rigorous about it. Even though we felt Xeon was unlikely to be our direction, we scheduled a meeting with Intel in Santa Clara (pre-COVID!) to understand their roadmap.

We wanted to take advantage of the all-day meeting to understand the other components that Intel was making that might be relevant for us, so we also asked for roadmap briefings on NIC silicon (Columbiaville) and switching silicon (Tofino). The host CPU discussion ended up confirming our beliefs (befitting our writing-intensive culture at Oxide we wrote up our findings in RFD 12 Host CPU Evaluation), and the NIC discussion similarly was a dead end. The switching silicon discussion, however, was interesting: Tofino was TSMC-fabbed (the only Intel part at the time fabbed outside of Intel) and we found the programmable nature of it via P4 to be really compelling. Up until that point, we had assumed that we would have to use Broadcom, and we weren’t thrilled at the prospects; the idea of having not just an alternative, but one with such a novel approach was hugely appealing. The Tofino team at Intel was enthused by our early interest, proving itself to be very responsive in the weeks that followed as we continued to explore the prospects of building around their Tofino 2. The more we learned, the more we liked!

As we got deeper with Tofino, we did, however have a single, substantial reservation: we didn’t trust Intel to not kill it. Intel has a long, long (long!) track record of fostering innovation outside of its mainstay x86 product — and then killing it. Making things more complicated, these untimely deaths are not without ambiguity: the efforts that Intel kills are often early and interesting — but need patience and more iterations to be able to win a broader market. And while the cullings are not always wrong (no amount of patience would have saved the iAPX 432), the biggest mistakes in the last two decades at Intel (namely, its failures in mobile CPUs and discrete GPUs) are a result of discarding a flawed effort entirely rather than learning from it and iterating.

Still, we loved the idea of P4 and programmable switch silicon, and we really liked the Tofino team at Intel: we felt our teams had a shared vision for the programmable data center — and they were very accommodating of our desire to build something very different from a commodity switch. So despite our trepidations, we stepped over the desiccated carcass of Red Rock Canyon, and engaged on building a switch with Tofino. And we didn’t (and don’t!) regret it: it was of course brutally hard to build our own switch, but the dividends of programmability were enormous — and thanks to Tofino and P4, we were able to build some important, unique functionality.

But by the time Gelsinger became CEO in early 2021, our finished product was still a bit in the future: we knew what we were building, but we were in the thick of building it. It was important for me to understand the incoming CEO, and if possible, to get an audience with Gelsinger to explain what we saw in Tofino (and to understand what his vision was for programmable networking silicon). So — and as I relayed in our recent podcast episode on Intel after Gelsinger — I tried to learn as much as possible about the guy in his own words. This would normally mean binge listening podcasts, talks, interviews, etc. — but in Gelsinger’s case, I hit an early and familiar jackpot: as part of their superlative series, the Computer History Museum had a four-hour oral history with Pat Gelsinger.

And in listening to Gelsinger, his affability clearly came across, along with his technical acumen and depth of experience. But something else came across too: an indisputable undercurrent of arrogance. Now, it’s a little hard to be critical, given the context: the whole point of an oral history after all is for people to speak about themselves. Still, these oral histories are most interesting when they take the opportunity to expand on failures, and Gelsinger’s narrative of his time at Intel seemingly just went from strength to strength. Worse, in his own story, the bad things at Intel always seemed to happen when he wasn’t in the room or otherwise over his objections — and the good things always when he was called in to save an effort from failure. I’m sure this retelling has truth to it, but I found it surprising that his narrative about AMD Opteron is of Intel conquering AMD with Nehalem under his leadership — with no mention of Intel’s gross architectural missteps with respect to 64-bit that allowed (and even demanded!) AMD Opteron in the first place.

But, fine. Again, it’s an oral history; there’s going to be a Rashomon effect. And so the guy was confident; so what? My feeling of his arrogance was still a bit amorphous when we got to Gelsinger retelling the end of his time at Intel. I was waiting for this, because I knew that Gelsinger had led the charge on Larrabee — and that Larrabee was infamously a wreck. He lamented leaving Intel because he knew that Larrabee would be killed after he departed (which it was), but I was shocked when he said that had Intel "stayed with it… NVIDIA would be a fourth the size they are today." I was washing the dishes as I was listening to this, and it stopped me in my tracks: I turned off the faucet, dried my hands, and backed up the recording. Had I heard correctly?! If one wanted to make this eyewatering claim, it must be loaded with riders and caveats: it must acknowledge that Larrabee itself was an unusable disaster; that NVIDIA had an indisputable lead, even in 2009; that for Intel to dominate NVIDIA it would have required conjuring software expertise and focus with which Intel has famously struggled; that Intel had no pattern for sustained success out of x86. On the one hand, there were enough qualifiers from Gelsinger to soften this claim a little (and at least some passing respect for NVIDIA’s Jenson Huang and Bill Dally), but on the other… yes, he actually claimed that NVIDIA had his departure from Intel to thank for (checks transcript) three quarters of its size. (Which, even in 2019 when the conversation was recorded, amounted to a whopping ~$75B!)

I think even the interviewers were a little taken aback, and asked the question directly: "Do you think that it was in part that you weren’t there to drive [Larrabee] forward that they decided to withdraw from that?" Gelsinger’s response: "Yeah, I’ll say that very directly."

This was beyond mere confidence, and was looking more like a total disregard for one’s own limitations. Another feeling was growing inside me: despite his many positive attributes, Gelsinger was the wrong person for the job. And to be clear, the job in 2021 was to lead a company in crisis, viz. the letter from Third Point that precipitated the firing of Bob Swan. Intel’s strategic mistakes were (in my opinion) symptomatic of an acute cultural problem: the company still carried with it the inherited arrogance from an earlier age. A concrete manifestation of the company’s arrogance is that it didn’t listen: it didn’t listen to its own people (and therefore struggled to correct course even when the rank-and-file knew that the trajectory is wrong) and it didn’t listen to its customers (and therefore built the wrong things for new markets — or missed out on those markets entirely). Intel needed a leader that could confront this cultural problem directly — who could work to undo an accretion of generations of entitlement — but if Gelsinger’s narrative for himself was any indicator, it felt like he would instead be feeding the company’s worst impulses about its own exceptionalism. As Gelsinger returned, I found no reassurance in the company’s narrative, which felt less like the arrival of Winston Wolf, and more like a cherished prince returning home to live out his destiny.

The concern that Gelsinger would be reliving his past more than navigating Intel’s future wasn’t exactly put to rest when, shortly after his arrival, Intel launched its cringy-as-hell anti-Apple "Go PC" ad campaign. (It tells you everything you need to know that Intel deleted the Go PC ads, but then forgot about Canada.) Fortunately for future historians of corporate dreck, the cringiest of these ads was immortalized when Marques Brownlee and David Imel absolutely ripped it apart. "Go PC" was an embodiment of the arrogance that I feared came from the top; how could anyone think that Intel’s biggest problem in 2021 was competing against… the Mac?! They knew that AMD also made x86 parts, right?! The whole campaign frankly felt juvenile, as if they were trying to just deride Apple for their decision to build their own silicon.

Much meatier than an ill-advised ad campaign was Gelsinger’s announcement that they would have 5 process nodes in 4 years. This was — and is — an aggressive bet for a company that had an entire process node (10nm) slip for years and then quietly fail to yield but a single product (Cannon Lake). From the perspective of an Intel customer, Intel had never come completely clean about the failing of 10nm (publicly or privately); how could Intel expect outsiders to trust them to learn from their mistakes if it wouldn’t even publicly acknowledge what those mistakes were? I don’t think that that bet was necessarily the wrong bet, but it was indisputably going to be expensive and risky. And this gets to Gelsinger’s first real, unequivocal mistake: he didn’t eliminate Intel’s dividend. Eliminating the dividend would have sent an early message to everyone — shareholders and employees — that the company’s survival was at stake, and it would have helped build the war chest for the battles to come. While I think his dichotomy is a bit reductive, Ben Horowitz’s nomenclature is useful here: Intel needed a wartime CEO — but in maintaining a dividend that it couldn’t afford, Gelsinger was committing the ultimate peacetime act. (I am not alone in this view; Ben Thompson of Stratechery dropped a must-listen episode on Intel in which he cites the paying out of billions of dollars of dividends over Gelsinger’s tenure as an early and serious blunder.)

Beyond being merely expensive and risky, the newly-christened Intel Foundry Services (IFS) had another, deeper problem: the culture at Intel didn’t really seem amenable to the level of customer engagement that foundry customers (rightfully) expect. And this was compounded by the trust issue that we at Oxide felt with Intel over Tofino: why would a customer trust their future to something chancy like IFS? Absent the availability of 18A, the only answer that Intel ever seemed to give on this was to wrap itself in the flag: that one should choose IFS solely because the fabs are on US soil. In this regard, it reminds me (ironically?) of a long-ago CPU startup, Transmeta. They had interesting technology, but unable to compete with Intel on absolute performance, they retreated to power efficiency. Transmeta was implicitly relying on the fact that Intel wouldn’t release a lower power part, but Intel promptly turned around and introduced the Pentium-M line — and that was more or less it for Transmeta. Similarly, is Intel assuming that TSMC is unable to build fabs in the US or that the US government wouldn’t see such fabs as addressing geopolitical concerns? If so, it feels like a bad assumption on both fronts.

Skepticism of Gelsinger’s plan for Intel aside, we at Oxide anxiously watched Tofino. At Intel, the team itself believed it was safe under Gelsinger, and things did indeed seem okay for a while. Fast-forward two years to 2023, and we got an urgent request for a call from the executive leading the Tofino effort. Fearing the worst, we were honestly somewhat relieved to learn that Tofino hadn’t been killed outright — but all future development of the part had been cancelled. We were on the cusp of shipping our first product, and the switch was working, so our own die was cast. And we were reassured by the Tofino team, who (to their great credit!) assured us that they would support us, even going to far as to draft a formal letter from Intel that we could share with our own customers expressing Intel’s ongoing support of Tofino.

While the situation for Tofino was obviously dire, I also felt we had some stoppage time; could we get in front of Gelsinger to explain why killing switching silicon in 2023 feels a lot like walking away from the GPGPU in 2011? No, as it turns out, we couldn’t. (And nor, apparently, could anyone else who was buying the part.) Everyone we dealt with at Intel agreed that the move was short-sighted, but as with so many things Intel (that culture problem again!), they felt powerless to change it.

Tofino was clearly living on borrowed time, and we were disappointed (though frankly not that surprised) when we were notified earlier this year that Tofino was being formally killed. It’s a credit to the Tofino team (or what was left of it at that point) that they continued to be very direct with us; this was (clearly) a decision that they disagreed with, and they were especially apologetic for the sloppy manner in which the end-of-life was being handled (which made a mockery of Intel’s own process for end-of-life management).

In the end, for all of the decisions that we made at Oxide — out of all of the companies and parts that we bet on, out of all the partners that we had sent RFD 68 Partnership as Shared Values to — only one had walked away from us, and it was the largest and best capitalized partner, who had repeatedly told us that they would not do exactly what they in fact did. How can Intel ever expected to be trusted when they treat partners this way?

For Oxide, fortunately, Intel might have done us a favor in the limit. While we were grateful to Tofino for allowing us to prove out our ideas on programmable networking, we had issues with it too. Specifically, Tofino’s software and instruction set architecture remained needlessly proprietary. We believe that switching silicon is awaiting its "x86 moment", when open source software can be implemented for a well-defined ISA — and we were never going to get there with Tofino. As we looked for our post-Tofino future, we were delighted to find the Xsight Labs team and their X2 ASIC. We are going to have a lot more to say about this part, but suffice it to say that we see in X2 both the strengths that we saw in Tofino and also the potential to be the open substrate for programmable networking writ large. Stay tuned!

I have great reverence for Intel and its extraordinary history, and I would never count them out (the resurrection of a clinically-dead AMD shows what is possible!), but I also won’t be integrating with any of their technology until their acute cultural issues are addressed. With regard to these cultural issues (and his other strengths aside), Pat Gelsinger was indisputably wrong for Intel.

dtrace.conf(24)

Thu, 05 Dec 2024 00:00:00 +0000

Sometime in late 2007, we had the idea of a DTrace conference. Or really, more of a meetup; from the primordial e-mail I sent:

The goal here, by the way, is not a DTrace user group, but more of a face-to-face meeting with people actively involved in DTrace — either by porting it to another system, by integrating probes into higher level environments, by building higher-level tools on top of DTrace or by using it heavily and/or in a critical role. That said, we also don’t want to be exclusionary, so our thinking is that the only true requirement for attending is that everyone must be prepared to speak informally for 15 mins or so on what they are doing with DTrace, any limitations that they have encountered, and some ideas for the future. We’re thinking that this is going to be on the order of 15-30 people (though more would be a good problem to have — we’ll track it if necessary), that it will be one full day (breakfast in the morning through drinks into the evening), and that we’re going to host it here at our offices in San Francisco sometime in March 2008.

This same note also included some suggested names for the gathering, including what in hindsight seems a clear winner: DTrace Bi-Mon-Sci-Fi-Con. As if knowing that I should leave an explanatory note to my future self as to why this name was not selected, my past self fortunately clarified: "before everyone clamors for the obvious Bi-Mon-Sci-Fi-Con, you should know that most Millennials don’t (sadly) get the reference." (While I disagree with the judgement of my past self, it at least indicates that at some point I cared if anyone got the reference.)

We settled on a much more obscure reference, and had the first dtrace.conf in March 2008. Befitting the style of the time, it was an unconference (a term that may well have hit its apogee in 2008) that you signed up to attend by editing a wiki. More surprising given the year (and thanks entirely to attendee Ben Rockwood), it was recorded — though this is so long ago that I referred to it as video taping (and with none of the participants mic’d, I’m afraid the quality isn’t very good). The conference, however, was terrific, viz. the reports of Adam, Keith and Stephen (all somehow still online nearly two decades later). If anything, it was a little too good: we realized that we couldn’t recreate the magic, and we demurred on making it an annual event.

Years passed, and memories faded. By 2012, it felt like we wanted to get folks together again, now under a post-lawnmower corporate aegis in Joyent. The resulting dtrace.conf(12) was a success, and the Olympiad cadence felt like the right one; we did it again four years later at dtrace.conf(16).

In 2020, we came back together for a new adventure — and the DTrace Olympiad was not lost on Adam. Alas, dtrace.conf(20) — like the Olympics themselves — was cancelled, if implicitly. Unlike the Olympics, however, it was not to be rescheduled.

More years passed and DTrace continued to prove its utility at Oxide; last year when Adam and I did our "DTrace at 20" episode of Oxide and Friends, we vowed to hold dtrace.conf(24) — and a few months ago, we set our date to be December 11th.

At first we assumed we would do something similar to our earlier conferences: a one-day participant-run conference, at the Oxide office in Emeryville. But times have changed: thanks to the rise of remote work, technologists are much more dispersed — and many more people would need to travel for dtrace.conf(24) than in previous DTrace Olympiads. Travel hasn’t become any cheaper since 2008, and the cost (and inconvenience) was clearly going to limit attendance.

The dilemma for our small meetup highlights the changing dynamics in tech conferences in general: with talks all recorded and made publicly available after the conference, how does one justify attending a conference in person? There can be reasonable answers to that question, of course: it may be the hallway track, or the expo hall, or the after-hours socializing, or perhaps some other special conference experience. But it’s also not surprising that some conferences — especially ones really focused on technical content — have decided that they are better off doing as conference giant O’Reilly Media did, and going exclusively online. And without the need to feed and shelter participants, the logistics for running a conference become much more tenable — and the price point can be lowered to the point that even highly produced conferences like P99 CONF can be made freely available. This, in turn, leads to much greater attendance — and a network effect that can get back some of what one might lose going online. In particular, using chat as the hallway track can be more much effective (and is certainly more scalable!) than the actual physical hallways at a conference.

For conferences in general, there is a conversation to be had here (and as a teaser, Adam and I are going to talk about it with Stephen O’Grady and Theo Schlossnagle on Oxide and Friends next week, but for our quirky, one-day, Olympiad-cadence dtrace.conf, the decision was pretty easy: there was much more to be gained than lost by going exclusively on-line.

So dtrace.conf(24) is coming up next week, and it’s available to everyone. In terms of platform, we’re going to try to keep that pretty simple: we’re going to use Google Meet for the actual presenters, which we will stream in real-time to YouTube — and we’ll use the Oxide Discord for all chat. We’re hoping you’ll join us on December 11th — and if you want to talk about DTrace or a DTrace-adjacent topic, we’d love for you to present! Keeping to the unconference style, if you would like to present, please indicate your topic in the #session-topics Discord channel so we can get the agenda fleshed out.

While we’re excited to be online, there are some historical accoutrements of conferences that we didn’t want to give up. First, we have a tradition of t-shirts with dtrace.conf. Thanks to our designer Ben Leonard, we have a banger of a t-shirt, capturing the spirit of our original dtrace.conf(08) shirt but with an Oxide twist. It’s (obviously) harder to make those free but we have tried to price them reasonably. You can get your t-shirt by adding it to your (free) dtrace.conf ticket. (And for those who present at dtrace.conf, your shirt is on us — we’ll send you a coupon code!)

Second, for those who can make their way to the East Bay and want some hangout time, we are going to have an après conference social event at the Oxide office starting at 5p. We’re charging something nominal for that too (and like the t-shirt, you pay for that via your dtrace.conf ticket); we’ll have some food and drinks and an Oxide hardware tour for the curious — and (of course?) there will be Fishpong.

Much has changed since I sent that e-mail 17 years ago — but the shared values and disposition that brought together our small community continue to endure; we look forward to seeing everyone (virtually) at dtrace.conf(24)!

Blogging through the decades

Sat, 16 Nov 2024 00:00:00 +0000

Over the summer, I hit an anniversary of sorts: I’ve been blogging for two decades (!). For reasons I’ll get to, I’ve been reflecting back on my history of writing in general and blogging in particular.

In 2004, blogging wasn’t exactly new (it had been around in one form or another for as long as a decade), but it also wasn’t something that I was actively engaged in. And while there was some blogging infrastructure at Sun, it was desultory and oriented around Java. Similarly, while there were also vectors for engineers to directly engage with customers, they were limited to proprietary forums, with very specific topics (e.g., support issues) and generally inaccessible and undiscoverable. All of that changed in the spring, when Sun rolled out a new policy on discourse and then – a few months later – blogs.sun.com. As Tim Bray retold, the disposition with respect to blogging was purposeful: employees weren’t merely allowed to blog, they were actively encouraged to do so – and provided all of the infrastructure to make it easy. The message was clear, and it was explicit: “We trust you.”

That trust – Sun’s finest quality! – was not new for the company; I came to work at Sun because I had seen an engineer (Jeff Bonwick) talk about their work (on comp.unix.solaris!), and I needed no convincing on the merits of technologist transparency. But wasn’t blogging just a little too… trendy? I quickly got over myself: we had a lot to talk about – DTrace had integrated in the fall of 2003 – and blogging felt like it was worth a shot. On June 14, 2004, I took the plunge. As it would be with Twitter Spaces many years later, it was immediately obvious that any reticence I might have had was misplaced – that blogging was larger than the sum of its parts. As I would describe to folks internally, the beauty of blogging is that you needn’t be regular and you also needn’t be on topic: you can write only when you have something to say – and you can say whatever you want.

Blogging at Sun was emblematic of a larger trend: the company that had invented open systems was in the process of reimagining itself in a new era of transparency and collaboration, as exemplified with the announced intention of open sourcing Solaris. While it had been announced earlier in 2004, open sourcing a complicated proprietary system like Solaris is deceptively tricky, and it was taking us a while to get all of the details right. Blogging was a perfect complement to our open sourcing activity; I blogged when DTrace went first in the chute – and when we ultimately open sourced the rest of the operating system later in 2005, we launched it by encouraging engineers to blog about it. When ZFS integrated into the system later that year, we marked the occasion with another flurry of blog entries. Thanks to blogging, we created an extraordinary amount of dense, technical content – and all in the voice of the technologists themselves.

The years went by, and as Tim Bray reflected back in 2007, blogging at Sun was essentially without downside. While I absolutely agree, it’s hard for me not to be a little bittersweet about that fifteen years later: there was so much great content that the only downside – such as it was – was that it was shackled to a listing corporate vessel. And when that vessel sank, much of the content went to a watery grave (or would have, had it not been for the heroics of the Internet Archive!).

Speaking for myself, when it came time to say good-bye to Sun, I didn’t want to leave my writing behind to its own fate. Fortunately, Sun had had the foresight to make clear to us that we ourselves owned our own writing, and I was able to easily export my data out of Sun to re-host it elsewhere. Even though I was going to work for a cloud provider I had not wanted to co-mingle my future job with my present one, so I did as one did: I created a WordPress blog on a third-party site. This WordPress site had the curse of being just good enough, and even after being at Joyent, moving it was never quite seemed worth it.

I continued to blog, though now with a cadence that was reduced due to the presence of microblogging (viz. our first two kids – born in 2004 and 2007 – were announced with blog posts; our third – born in 2012 – with a tweet). Having a new vector for hot takes meant that blogging could be reserved for long-form writing (or, occasionally, long-form hot takes); when we released something new at Joyent – like KVM, Manta, or Triton – I used blogging to tell some of the human side of the story.

Years passed. Joyent was bought by Samsung. I fell in love with Rust. I left Joyent and started Oxide. At Oxide, long-form writing remained important as ever, and I used blogging to not just talk about our technologies (like Hubris and Humility) but also to talk about why we were building the company the way were building it, from specifics like our approach to compensation to our more general thoughts on engineering a culture.

While blogging continued to be important, that hosted WordPress lifeboat that I scrambled aboard in 2010 was becoming really problematic; it would constantly spring new leaks and was rife with security vulnerabilities – and holy moly was it slow! I knew I needed to move my blog to a better spot, but could never quite prioritize it…

Into this stasis, I was approached by Piotr Sarna and Cynthia Dunlop, who were working on a new book, Writing for Developers. They had included references to two of my pieces in the book (Rust after the honeymoon and The relative performance of C and Rust), and they wanted me to write the foreword. Excited that they were taking on this topic and honored that they would ask me to write it, I agreed. In preparation for the launch of the book, Cynthia sent me some thoughtful questions about my own blogging, wondering if I might be willing to blog my answers. Anyone who has seen the Oxide hiring process knows that I love reflective questions, and Cynthia asked me some great ones: What blog entry was I most proud of? Which one was most difficult to write? What impact of blogging have I found surprising? What advice would I have for those getting started with blogging? Of course, answering these questions required me to go back and read my older blog entries – and it is with this that I (finally) hit my breaking point with WordPress: if reading a single blog entry was painful, trying to navigate through ~150 of them over twenty years was untenable. The lifeboat had served its duty, but it was now time to beach it and turn it into kindling: to answer Cynthia’s questions, I had to first find the blog a new home.

A lot had changed since 2004, and even since my WordPress lifeboat in 2010. For my blog, I wanted to get away from systems that feel like unnecessarily complicated content management systems. Fortunately, static site generators represent the approach that I wanted from a blog: blazingly fast, clean, readable on mobile, and (most importantly for someone who was fleeing captive content management for the second time), entirely git-backed. There were some good options to choose from, but I landed on Hugo, using the hugo-blog-awesome theme – and Will Boyd’s excellent wordpress-export-to-markdown for facilitating the migration.

In addition to the rise of static site generators, there of course had been another big change: we shipped the Oxide cloud computer! To allow customers to try out the Oxide rack without needing to first buy one, we stood up an Oxide rack in a datacenter environment, spinning up a silo for folks who want to kick the tires. And increasingly, we use this production rack for our own infrastructure at Oxide. All of this gave me an opportunity for a special homecoming for my blog: it could run on the system that we have built from scratch! A blog may be the simplest possible use case for the Oxide rack, but it was still really fun to use the product – and (huge credit to the folks building the frontend of the computer) so damned easy. I especially loved how easy it was to do VPC management, which quickly becomes complicated.

All of this to say: the text you are reading now, dear reader, is being served from a web server running on an instance in a production Oxide rack!

Having now decades of blogging readily accessible, it would be easy to take on Cynthia’s questions, right? Well, not so fast. It’s indeed easy to read all of my writing, but the more I read, the more ambiguity there was. My decades of blogging were teeming with life: birth (Fishworks, Manta, Oxide, illumos), love (JavaScript, systems, Rust, The Soul of a New Machine) and death (Sun, Solaris). And the death, sadly, was not merely metaphorical; over the years, I memorialized the passing of three people who still had a lot of life in front of them: John Birrell, Khaled Bichara, and – tragically recently – Charles Beeler.

I have written some very dense technical pieces and taken on some scalding hot topics; I have walked through history’s graveyard, pondered the quality of software, and reflected on its economics. The throughline of two decades of writing, especially for the pieces that have enduring resonance: they reflect the humanity in our endeavor. There is an old adage to “speak from the heart, not from the book”; looking back, I am a bit surprised at the degree that this was true in my own writing. And while I don’t have neat answers to Cynthia’s questions, it does leave me with easy advice for would-be bloggers: write from the heart, even if you think no one is reading it; if nothing else, your future self will thank you for it!

Remembering Charles Beeler

Tue, 12 Nov 2024 00:00:00 +0000

Note: This was co-authored with Steve Tuck, and originally appeared on the Oxide blog.

We are heartbroken to relay that Charles Beeler, a friend and early investor in Oxide, passed away in September after a battle with cancer. We lost Charles far too soon; he had a tremendous influence on the careers of us both.

Our relationship with Charles dates back nearly two decades, to his involvement with the ACM Queue board where he met Bryan. It was unprecedented to have a venture capitalist serve in this capacity with ACM, and Charles brought an entirely different perspective on the practitioner content. A computer science pioneer who also served on the board took Bryan aside at one point: "Charles is one of the good ones, you know."

When Bryan joined Joyent a few years later, Charles also got to know Steve well. Seeing the promise in both node.js and cloud computing, Charles became an investor in the company. When companies hit challenging times, some investors will hide — but Charles was the kind of investor to figure out how to fix what was broken. When Joyent needed a change in executive leadership, it was Charles who not only had the tough conversations, but led the search for the leader the company needed, ultimately positioning the company for success.

Aside from his investment in Joyent, Charles was an outspoken proponent of node.js, becoming an organizer of the Node Summit conference. In 2017, he asked Bryan to deliver the conference’s keynote, but by then, the relationship between Joyent and node.js had become… complicated, and Bryan felt that it probably wouldn’t be a good idea. Any rational person would have dropped it, but Charles persisted, with characteristic zeal: if the Joyent relationship with node.js had become strained, so much more the reason to speak candidly about it! Charles prevailed, and the resulting talk, Platform as Reflection of Values, became one of Bryan’s most personally meaningful talks.

Charles’s persistence was emblematic: he worked behind the scenes to encourage people to do their best work, always with an enthusiasm for the innovators and the creators. As we were contemplating Oxide, we told Charles what we wanted to do long before we had a company. Charles laughed with delight: "I hoped that you two would do something big, and I am just so happy for you that you’re doing something so ambitious!"

As we raised seed capital, we knew that we were likely a poor fit for Charles and his fund. But we also knew that we deeply appreciated his wisdom and enthusiasm; we couldn’t resist pitching him on Oxide. Charles approached the investment in Oxide as he did with so many other aspects: with curiosity, diligence, empathy, and candor. He was direct with us that despite his enthusiasm for us personally, Oxide would be a challenging investment for his firm. But he also worked with us to address specific objections, and ultimately he won over his partnership. We were thrilled when he not only invested, but pulled together a syndicate of like-minded technologists and entrepreneurs to join him.

Ever since, he has been a huge Oxide fan. Befitting his enthusiasm, one of his final posts expressed his enthusiasm and pride in what the Oxide team has built.

Charles, thank you. You told us you were proud of us — and it meant the world. We are gutted to no longer have you with us; your influence lives on not just in Oxide, but also in the many people that you have inspired. You were the best of venture capital. Closer to the heart, you were a terrific friend to us both; thank you.

Reflections on Founder Mode

Mon, 02 Sep 2024 00:00:00 +0000

Paul Graham’s Founder Mode is an important piece, and you should read it if for no other reason that “founder mode” will surely enter the lexicon (and as Graham grimly predicts: “as soon as the concept of founder mode becomes established, people will start misusing it”). When building a company, founders are engaged in several different acts at once: raising capital; building a product; connecting that product to a market; building an organization to do all of these. Founders make lots of mistakes in all of these activities, and Graham’s essay highlights a particular kind of mistake in which founders are overly deferential to expertise or convention. Pejoratively referring to this as “Management Mode”, Graham frames this in the Silicon Valley dramaturgical dyad of Steve Jobs and John Scully. While that’s a little too reductive (anyone seeking to understand Jobs needs to read Randall Stross’s superlative Steve Jobs and the NeXT Big Thing, highlighting Jobs’s many post-Scully failures at NeXT), Graham has identified a real issue here, albeit without much specificity.

For a treatment of the same themes but with much more supporting detail, one should read the (decade-old) piece from Tim O’Reilly, How I failed. (Speaking personally, O’Reilly’s piece had a profound influence on me, as it encouraged me to stand my ground on an issue on which I had my own beliefs but was being told to defer to convention.) But as terrific as it is, O’Reilly’s piece also doesn’t answer the question that Graham poses: how do founders prevent their companies from losing their way?

Graham says that founder mode is a complete mystery (“There are as far as I know no books specifically about founder mode”), and while there is a danger in being too pat or prescriptive, there does seem to be a clear component for keeping companies true to themselves: the written word. That is, a writing- (and reading-!) intensive company culture does, in fact, allow for scaling the kind of responsibility that Graham thinks of as founder mode. At Oxide, our writing-intensive culture has been absolutely essential: our RFD process is the backbone of Oxide, and has given us the structure to formalize, share, and refine our thinking. First among this formalized thinking – and captured in our first real RFD – is RFD 2 Mission, Principles, and Values. Immediately behind that (and frankly, the most important process for any company) is RFD 3 Oxide Hiring Process. These first three RFDs – on the process itself, on what we value, and on how we hire – were written in the earliest days of the company, and they have proven essential to scale the company: they are the foundation upon which we attract people who share our values.

While the shared values have proven necessary, they haven’t been sufficient to eliminate the kind of quandaries that Graham and O’Reilly describe. For example, there have been some who have told us that we can’t possibly hire non-engineering roles using our hiring process – or told us that our approach to compensation can’t possibly work. To the degree that we have had a need for Graham’s founder mode, it has been in those moments: to stay true to the course we have set for the company. But because we have written down so much, there is less occasion for this than one might think. And when it does occur – when there is a need for further elucidation or clarification – the artifact is not infrequently a new RFD that formalizes our newly extended thinking. (RFD 68 is an early public and concrete example of this; RFD 508 is a much more recent one that garnered some attention.)

Most importantly, because we have used our values as a clear lens for hiring, we are able to assure that everyone at Oxide is able to have the same disposition with respect to responsibility – and this (coupled with the transparency that the written word allows) permits us to trust one another. As I elucidated in Things I Learned The Hard Way, the most important quality in a leader is to bind a team with mutual trust: with it, all things are possible – and without it, even easy things can be debilitatingly difficult. Graham mentions trust, but he doesn’t give it its due. Too often, founders focus on the immediacy of a current challenge without realizing that they are, in fact, undermining trust with their approach. Bluntly, founders are at grave risk of misinterpreting Graham’s “Founders Mode” to be a license to micromanage their teams, descending into the kind of manic seagull management that inhibits a team rather than empowering it.

Founders seeking to internalize Graham’s advice should recast it by asking themselves how they can foster mutual trust – and how they can build the systems that allow trust to be strengthened even as the team expands. For us at Oxide, writing is the foundation upon which we build that trust. Others may land on different mechanisms, but the goal of founders should be the same: build the trust that allows a team to kick a Jobsian dent in the universe!

Engineering a culture

Sun, 31 Mar 2024 00:00:00 +0000

We ran into an interesting issue recently. On the one hand, it was routine: we had a bug — a regression — and the team quickly jumped on it, getting it root caused and fixed. But on the other, this particular issue was something of an Oxide object lesson, representative not just of the technologies but also of the culture we have built here. I wasn’t the only person who thought so, and two of my colleagues wrote terrific blog entries with their perspectives:

Matt Keeter with It’s Free Real Estate
Cliff Biffle with Who killed the network switch? A Hubris Bug Story

The initial work as described by Matt represents a creative solution to a thorny problem; if it’s clear in hindsight, it certainly wasn’t at the time! (In Matt’s evocative words: "One morning, I had a revelation.") I first learned of Matt’s work when he demonstrated it during our weekly Demo Friday, an hour-long unstructured session to demo our work for one another. Demo Friday is such an essential part of Oxide’s culture that it feels like we have always done it, but in fact it took us nearly two years into the company’s life to get there: over the spring and summer of 2021, our colleague Sean Klein had instituted regular demos for the area that he works on (the Oxide control plane), and others around the company — seeing the energy that came from it — asked if they, too, could start regular demos for their domain. But instead of doing it group by group, we instituted it company-wide starting in the fall of 2021: an unstructured hour once a week in which anyone can demo anything.

In the years since, we have had demos of all scopes and sizes. Importantly, no demo is too small — and we have often found that a demo that feels small to someone in the thick of work will feel extraordinary to someone outside of it. ("I have a small demo building on the work of a lot of other people" has been heard so frequently that it has become something of an inside joke.) Demo Friday is important because it gets to one of our most important drivers as technologists: the esteem of our peers. The thrill that you get from showing work to your colleagues is unparalleled — and their wonderment in return is uniquely inspiring. (Speaking personally, Matt’s demo addressed a problem that I had personally had many times over in working on Hubris — and I was one of the many w00ts in the chat, excited to see his creative solution!)

Having the demos be company-wide has also been a huge win for not just our shared empathy and teamwork but also our curiosity and versatility: it’s really inspiring to have (say) one colleague show how they used PCB backdrilling for signal integrity, and the next show an integration they built using Dropshot between our CRM and spinning up a demonstration environment for a customer. And this is more than just idle intellectual curiosity: our stack is deep — spanning both hardware and software — and the demos make for a fun and engaging way to learn about aspects of the system that we don’t normally work on.

Returning to Matt and Cliff, if Matt’s work implicitly hits on aspects of our culture, Cliff’s story of debugging addresses that culture explicitly, noting that the experience demonstrated:

Tight nonhierarchical integration of the team. This isn’t a Hubris feature, but it’s hard to separate Hubris from the team that built it. Oxide’s engineering team has essentially no internal silos. Our culture rewards openness, curiosity, and communication, and discourages defensiveness, empire-building, and gatekeeping. We’ve worked hard to create and defend this culture, and I think it shows in the way we organized horizontally, across the borders of what other organizations would call teams, to solve this mystery.

In the discussion on Hacker News of Cliff’s piece, this cultural observeration stood out, with a commenter asking:

I’d love to hear more about the motivations for crafting such a culture as well as some particular implementation details. I’m curious if there are drawbacks to fostering "openness, curiosity, and communication" within an organization?

The culture at Oxide is in fact very deliberate: when starting a company, one is building many things at once (the team, the product, the organization, the brand) — and the culture will both inform and be reinforced by all of these. Setting that first cultural cornerstone was very important to us — starting with our mission, principles, and values. Critically, by using our mission, principles, and values as the foundation for our hiring process, we have deliberately created a culture that reinforces itself.

Some of the implementation details:

We have uniform compensation (even if it might not scale indefinitely)
We are writing intensive (but we still believe in spoken collaboration)
We have no formalized performance review process (but we believe in feedback)
We record every meeting (but not every conversation)
We have a remote work force (but we also have an office)
We are non-hierarchical (but we all ultimately report to our CEO)
We don’t use engineering metrics (but we all measure ourselves by our customers and their success)

If it needs to be said, there is plenty of ambiguity: if you are using absolutes to think of Oxide (outside of our principles of honesty, integrity and decency!) you are probably missing some nuance of our culture.

Finally, to the (seemingly loaded?) question of the "drawbacks" of fostering "openness, curiosity, and communication" within an organization, the only drawback is that it’s hard work: culture has to be deliberate without being overly prescriptive, and that can be a tricky balance. In this regard, building a culture is very different than building (say) software: it is not engineered in a traditional sense, but is rather a gooey, squishy, organism that will evolve over time. But the reward of the effort is something that its participants care intensely about: it will continue to be (in Cliff’s words) a culture that we work hard to not just create but defend!

Moore’s scofflaws

Tue, 20 Feb 2024 00:00:00 +0000

Years ago, Jeff Bezos famously quipped that "your margin is my opportunity." This was of course aimed not at Amazon’s customers, but rather its competitors, and it was deadly serious: customers of AWS in those bygone years will fondly remember that every re:Invent brought with it another round of price cuts. This era did not merely reflect Bezos’s relentless execution, but also a disposition towards who should reap the reward of advances in underlying technology: Amazon believed (if implicitly) that improvements at the foundations of computing (e.g., in transistor density, core count, DRAM density, storage density, etc.) should reflect themselves in lower prices for consumers rather than higher margins for suppliers.

Price cuts are no longer a re:Invent staple, having been replaced by a regular Amazon tradition of a different flavor: cutting depreciation (and therefore increasing earnings) by extending the effective life of their servers. (These announcements are understandably much more subdued, as "my depreciation is my margin opportunity" doesn’t have quite the same ring to it.)

As compute needs have grown and price cuts have become an increasingly distant memory, some have questioned their sky-high cloud bills, wondering if they should in fact be owning their compute instead of renting it. When we started Oxide, we knew from operating our own public cloud what those economics looked like — and we knew that over time others of a particular scale would come to the same realization that they would be better off not giving their margin away by renting compute. (Though it’s safe to say that we did not predict that it would be DHH leading the charge!)

Owning one’s own cloud sounds great, but there is a bit that’s unsaid: what about the software? Software is essential for elastic, automated infrastructure: hardware alone does not a cloud make! Unfortunately, the traditional server vendors do not help here: because of a PC-era divide in how systems are delivered, customers are told to look elsewhere for any and all system software. This divide is problematic on several levels. First, it impedes the hardware/software co-design that we (and, famously, others!) believe is essential to deliver the best possible product. Second, it leads to infamous finger pointing when the whole thing doesn’t work. But there is also a thorny economic problem: when your hardware and your software don’t come from the same provider, to whom should go the spoils of better hardware?

To someone who has just decided to buy their hardware out of their frustration with renting it, the answer feels obvious: whoever owns the hardware should naturally benefit from its advances! Unfortunately, the enterprise software vendor delivering your infrastructure often has other ideas — and because their software is neither rented nor bought, but rather comes from the hinterlands of software licensing, they have broad latitude as to how it is priced and used. In particular, this allows them to charge based on the hardware that you run it on — to have per-core software licensing.

This galling practice isn’t new (and is in fact as old as symmetric multiprocessing systems), but it has taken on new dimensions in the era of chiplets and packaging innovation: the advances that your next CPU has over your current one are very likely to be expressed in core count. Per-core licensing allows a third party — who neither made the significant investment in developing the next generation of microprocessor nor paid for the part themselves — to exact a tax on improved infrastructure. (And this tax can be shockingly brazen!) Couple this with the elimination of perpetual licensing, and software costs can potentially absorb the entire gain from a next-generation CPU, leaving a disincentive to run newer, more efficient infrastructure. As an industry, we have come to accept this practice, but we shouldn’t: in the go-go era of Dennard scaling (when clock rates rose at a blistering rate), software vendors never would have been allowed to get away with charging by the gigahertz; we should not allow them to feel so emboldened to charge by core count now!

If it needs to be said, we have taken a different approach at Oxide: when you buy the Oxide cloud computer, all of the software to run it is included. This includes all of the software necessary to run the rack as elastic infrastructure: virtual compute, virtual storage, virtual networking. (And yes, it’s all open source — which unfortunately demands the immediate clarification that it’s actually open source rather than pretend open source.) When we add a new feature to our software, there is no licensing enablement or other such nuisance — the feature just comes with the next update. And what happens when AMD releases a new CPU with twice the core count? The new sled running the new CPU runs along your existing rack — you’re not paying more than the cost of the new sled itself. This gives the dividends of Moore’s Law (or Wright’s Law!) to whom they rightfully belong: the users of compute.

What punch cards teach us about AI risk

Sun, 26 Nov 2023 00:00:00 +0000

I (finally) read Edwin Black’s IBM and the Holocaust, and I can’t recommend it strongly enough. This book had been on my queue for years, and I put it off for the same reason that you have probably put it off: we don’t like to confront difficult things. But the book is superlative: not only is it fascinating and well-researched but given the current level of anxiety about the consequences of technological development, it feels especially timely. Black makes clear in his preface that IBM did not cause the Holocaust (unequivocally, the Holocaust would have happened without IBM), but he also makes clear in the book that information management was essential to every aspect of the Nazi war machine – and that that information management was made possible through IBM equipment and (especially) their punch cards.

I have known little of computing before the stored program computer, and two aspects of punch card systems of this era were surprising to me: first, to assure correct operation in these most mechanical of systems, the punch cards themselves must be very precisely composed, manufactured, and handled – and the manufacturing process itself is difficult to replicate. Second, punch cards of this era were essentially single-use items: once a punch card had been through a calculation, it had to be scrapped. Given that IBM was the only creator of punch cards for its machines, this may sound like an early example of the razor blade model, but it is in fact even more lucrative: IBM didn’t sell the machines at a discount because they didn’t sell the machines at all – they rented them. This was an outrageously profitable business model, and a reflection of the most dominant trait of its CEO, Thomas J. Watson: devotion to profit over all else.

In the Nazis, Watson saw a business partner to advance that profit – and they saw in him an American advocate for appeasement, with Hitler awarding Watson its highest civilian medal in 1937. (In this regard, the Nazis themselves didn’t understand that Watson cared only about profit: unlike other American Nazi sympathizers, Watson would support an American war effort if he saw profit in it – and he publicly returned the medal after the invasion of Holland in 1940, when public support of the Nazis had become a clear commercial liability.) A particularly revealing moment with respect to Watson’s disposition was in September 1939 (after the invasion of Poland!) when IBM’s German subsidiary (known at the time as Dehomag) made the case to him that the IBM 405 alphabetizers owned by IBM’s Austrian entity in the annexed Austria now belonged to the German entity to lease as they please. These particular alphabetizers were important: the 405 was an order of magnitude improvement over the IBM 601 – and it was not broadly found in Europe. Watson resisted handing over the Austrian 405s, though not over any point of principle, but rather of avarice: in exchange for the 405s, he demanded (as he had throughout the late 1930s) that he have complete ownership of IBM’s German subsidiary rather than the mere 90% that IBM controlled. The German subsidiary refused the demand and ultimately Watson relented – and the machines effectively became enlisted as German weapons of war.

IBM has made the case that it did not know how its machines were used to effect the Holocaust, but this is hard to believe given Watson’s level of micromanagement of the German subsidiary through Switzerland during the war: IBM knew which machines were where (and knew, for example, that concentration camps all had ample sorters and tabulators), to the point that the company was able to retrieve them all after the war – along with the profits that the machines had earned.

This all has much to teach us about the present day with respect to the true risks of technology. Technology serves as a force-multiplier on humanity, for both better and ill. The most horrific human act – genocide – requires organization and communication, two problems for which we have long developed technological solutions. Whether it was punch cards and tabulators in the Holocaust, radio transmission in the Rwandan Genocide, or Facebook in the Rohingya genocide, technology has sadly been used as an essential tool for our absolute worst. It may be tempting to blame the technology itself, but that in fact absolves the humans at the helm. Should we have stymied the development of tabulators and sorters in the 1920s and 1930s? No, of course not. And nor, for that matter, should Rwanda have been deprived of radio or Myanmar of social media. But this is not to say that we should ignore technology’s role, either: the UN erred in not destroying the radio transmission capabilities in Rwanda; Facebook erred by willfully ignoring the growing anti-Rohingya violence; and IBM emphatically erred by being willing to supply the Nazis in the name of its own profits.

To bring this into the present day: as I relayed in my recent Monktoberfest talk, the fears of AI autonomously destroying humanity are worse than nonsense, because they distract us from the very real possibilities of how AI may be abused. To allow ourselves to even contemplate a prohibition of the development of certain kinds of computer programs is to delude ourselves into thinking that the problem is a technical problem rather than a human one. Worse, the very absurdity of prohibition has itself created a reactionary movement in the so-called “effective accelerationists” who, like some AI equivalent of rolling coal, refuse to contemplate any negative ramifications of technological development whatsoever. This, too, is grievously wrong, and we need look no further than IBM’s involvement in the Holocaust to see the peril of absolute adherence to technology-based profit.

So what course to chart with respect to the (real, human) risks of AI? We should consider another important fact of IBM’s involvement with the Nazis: IBM itself skirted the law. Some the most interesting findings in Black’s book are from the US Department of Treasury’s 1943 investigation into IBM’s collusion with Hitler. The investigator – Harold Carter – had plenty of evidence that IBM was violating the Trading with the Enemy Act, but Watson had also so thoroughly supported the Allied war effort that he was unassailable within the US. We already have regulatory regimes with respect to safety: you can’t just obtain fissile material or make a bioweapon – it doesn’t matter if ChatGPT told you to do it or not. We should be unafraid to enforce existing laws. Believing that (say) Uber was wrong to illegally put their self-driving cars on the street does not make one a “decel” or whatever – it makes one a believer in the rule of law in a democratic society. That this sounds radical – that one might believe in a democracy that creates laws, affords companies economic freedom within those laws, and enforces those laws against companies that choose to violate them – says much about our divisive times.

And all of this brings us to the broadest lesson of IBM and the Holocaust: technological development is by its nature new – a lurch into the unknown and unexplored – but as I have discovered over and over again in my career, history has much to teach us. Even though the specifics of the technologies we work on may be without precedent, the humanity they serve to advance endures across generations; those who fret about the future would be well advised to learn from the past!

Is it worse for John Fisher?

Sat, 18 Nov 2023 00:00:00 +0000

“It’s been worse for me than for you.” These extraordinary words came out of the mouth of John Fisher, incompetent owner of the Oakland Athletics, on the eve of getting approval from Major League Baseball to rip its roots out of the East Bay.

I have been reflecting a lot on these words. Strictly from a public relations point of view, they are gobsmackingly disrespectful, plumbing new depths of malpractice even for the worst ownership in sports. And of course, they are obviously wrong, as this clumsy move is worse for literally everyone else than it is for John Fisher. It is worse for the fans having their hearts ripped out; worse for the Oakland employees losing their jobs; worse for the many small businesses that make their livelihood on the team; worse for the players who have been told their entire athletic careers to take accountability only to be forced to watch in silence as their skinflint ownership takes none.

But there is a kind of truth to these words too, in that there are ways that it is worse for Fisher, for we have things that he cannot. Take, for example, the Reverse Boycott, the game on June 13th, 2023 when Oakland fans deliberately attended to show that we are, in fact, not the problem. Everything about that game was extraordinary: the energy was post-season electric as the worst-in-baseball A’s entered the game with a best-in-baseball win streak. The Coliseum was rocking, in a way that only the Coliseum can. Then, at the top of the 5th inning, the fans fell silent in protest of the move to Las Vegas. There was no plan beyond this; no one really knew what would happen when the silence ended. What happened next was spontaneous, straight from a shared heart that was breaking: a deafening chant, rolling and crashing over the stadium. “SELL! THE! TEAM! SELL! THE! TEAM!” (I accidentally recorded this; you can hear the emotion in my own voice – and that of my 11-year-old daughter next to me.) The game ended as only fiction would have it: with Trevor May striking out the best team in baseball to seal an improbable win for Oakland. The biggest surprise of the night was the sheer joy of it all: it was a New Orleans funeral for Oakland baseball, and we were glad to be there as a family. As I told my kids on the drive home, it was a night that they would one day tell their own grandchildren about.

How is it that a baseball game can conjure such emotion, let alone one from a losing franchise with a signed death warrant? Because, simply: sports are about much more than what’s on the field. Sports bring us together – they bind us across generation, disposition, and circumstance. A family that might agree on little else may shout in indignant agreement that that wasn’t pass interference or that he was obviously safe. They give us solidarity with one another: they give us stuff to believe in together, to celebrate together – and to grieve for together. In short, sports are the raw id of our own humanity. The Reverse Boycott distilled all of it into a single, singular night – binding us together in the kind of shared adversity that has always been the stuff of tribal legend.

And it is in this regard that John Fisher might be right: it is, in fact worse for him, because this shared humanity of sports eludes him. His camera roll is not filled with A’s-themed birthday parties, or of selfies with his kids in rally caps, or of toddlers running the bases late on a Sunday afternoon. It would be tempting to say that he instead sees sports as only a business, but even this gives him too much credit: the only business he knows is assuring the mechanics of inheritance – of hoarding the spoils of his birth. In this regard, he is at least passably capable: he took MLB at its word that it would cut off his welfare payments if he did not secure a stadium deal by January 2024, and dutifully secured a deal, however obviously disastrous. It’s worse for John Fisher because this has all been laid bare: the real cost of securing his allowance is that his ineptitude is no longer merely an open secret among beleaguered A’s fans – he is now MLB’s famous failson, the Connor Roy of professional sports.

Whatever success John Fisher may find in Las Vegas, he will not be able to outrun the wreckage he is leaving behind here in Oakland. In John Fisher’s obituary, it will not speak of what he built, but of what he broke; not of what he gave, but of what he took away. He will be a stain on his family, who will spend their lives trying to apologize for him. He himself will find that no amount of success will absolve him of the scar that he is leaving on the East Bay’s heart. And the much more likely scenario – abject commercial failure – will merely confirm for him his own nightmares: that he is exactly the klutz and dunce that he surely fears himself to be. And if John Fisher will always be searching for what he cannot get, we Oakland A’s fans will always have what cannot be taken away: our solidarity with – and love for – one another. We are raucous, brainy, creative, and eclectic; our lives are richer for having one another in them. John Fisher has none of this, and never will. As terrible as it is for us, it may indeed be worse for him.

The Cloud Computer

Thu, 26 Oct 2023 00:00:00 +0000

Today we are announcing the general availability of the world’s first commercial cloud computer — along with our $44M Series A financing.

From the outset at Oxide, and as I outlined in my 2020 Stanford talk, we have had three core beliefs as a company:

Cloud computing is the future of all computing infrastructure.
The computer that runs the cloud should be able to be purchased and not merely rented.
Building a cloud computer necessitates a rack-level approach — and the co-design of both hardware and software.

Of these beliefs, the first is not at all controversial: the agility, flexibility, and scalability of cloud computing have been indisputably essential for many of the services that we depend on in the modern economy.

The degree that the second belief is controversial, however, depends on who you are: for those that are already running on premises due to security, regulatory, economic, or latency reasons, it is self-evident that computers should be able to be purchased and not merely rented. But to others, this has been more of a revelation — and since we started Oxide, we have found more and more people realize that the rental-only model for the cloud is not sustainable. Friends love to tag us on links to VC thinkpieces, CTO rants, or analyst reports on industry trends — and we love people thinking of us, of course (even when being tagged for the dozenth time!) — but the only surprise is how surprising it continues to be for some folks.

The third belief — that the development of a cloud computer necessitates rack-scale design of both hardware and software — may seem iconoclastic to those who think only in terms of software, but it is in fact not controversial among technologists: as computing pioneer Alan Kay famously observed, "people who are really serious about software should make their own hardware." This is especially true in cloud computing, where the large public cloud companies have long ago come to the conclusion that they needed to be designing their own holistic systems. But if this isn’t controversial, why hasn’t there been a cloud computer before Oxide’s? First, because it’s big: to meaningfully build a cloud computer, one must break out of the shackles of the 1U or 2U server, and really think about the rack as the unit of design. Second, it hasn’t been done because it’s hard: co-designing hardware and software that spans compute, networking, and storage requires building an extraordinary team across disparate disciplines, coupling deep expertise with a strong sense of versatility, teamwork, and empathy. And the team isn’t enough by itself: it also needs courage, resilience, and (especially) time.

So the biggest question when we set out was not "is the market there?" or "is this the right way to do it?", but rather could we pull this off?

Pulling it off

We have indeed pulled it off — and it’s been a wild ride! While we have talked about the trek quite a bit on our podcast, Oxide and Friends (and specifically, Steve and I recently answered questions about the rack), our general availability is a good opportunity to reflect on some of the first impressions that the Oxide cloud computer has made upon those who have seen it.

"Where are all the boxes?"

The traditional rack-and-stack approach starts with a sea of boxes arriving with servers, racks, cabling, etc. This amounts to a literal kit car approach — and it starts with tedious, dusty, de-boxing. But the Oxide rack ships with everything installed and comes in just one box — a crate that is its own feat of engineering. All of this serves to dramatically reduce the latency from equipment arrival to power on and first provision — from weeks and months to days or even hours.

"Is it on?"

We knew at the outset that rack-level design would afford us the ability to change the geometry of compute sleds — that we would get higher density in the rack by trading horizontal real estate for vertical. We knew, too, that we were choosing to use 80mm fans for their ability to move more air much more efficiently — so much so that we leveraged our approach to the supply chain to partner with Sanyo Denki (our fan provider) to lower the minimum speed of the fans from 5K RPM to the 2K RPM that we needed. But adding it up, the Oxide rack has a surprising aesthetic attribute: it is whisper quiet. To those accustomed to screaming servers, this is so unexpected that when we were getting FCC compliance, the engineer running the test sheepishly asked us if we were sure the rack was on — when it was dissipating 15 kW! That the rack is quiet wasn’t really deliberate (and we are frankly much more interested in the often hidden power draw that blaring fan noise represents), but it does viscerally embody much of the Oxide differentiation with respect to both rack-level design and approach to the supply chain.

"Where are the cables?"

Anyone accustomed to a datacenter will note the missing mass of cold-aisle cabling that one typically sees at the front of a rack. But moving to the back of the rack reveals only a DC busbar and a tight, cabled backplane. This represents one of the bigger bets we made: we blindmated networking. This was mechanically tricky, but the payoff is huge: capacity can be added to the Oxide cloud computer simply by snapping in a new compute sled — nothing to be cabled whatsoever! This is a domain in which we have leapfrogged the hyperscalers, who (for their own legacy reasons) don’t do it this way. This can be jarring to veteran technologists. As one exclaimed upon seeing the rack last week, "I am both surprised and delighted!" (Or rather: a very profane variant of that sentiment.)

"You did your own switch too?!"

When we first started the company, one of our biggest technical quandaries was what to do about the switch. At some level, both paths seemed untenable: we knew from our own experience that integrating with third-party switches would lead to exactly the kind of integration pain for customers that we sought to alleviate — but it also seemed outrageously ambitious to do our own switch in addition to everything else we were doing. But as we have many times over the course of Oxide, we opted for the steeper path in the name of saving our customers grief, choosing to build our own switch. If it has to be said, getting it working isn’t easy! And of course, building the switch is insufficient: we also needed to build our own networking software — to say nothing of the management network required to be able to manage compute sleds when they’re powered off.

"Wait, that’s part of it?!"

It’s one thing to say that all of the software that one needs to operate the cloud computer is built in — but it’s another to actually see what that software includes. And for many, it’s seeing the Oxide web console (or its live demo!) that really drives the message home: yes, all of the software is included. And because the console implementation is built on the public API, everything that one can do in the console for the Oxide rack is also available via CLI and API — a concrete manifestation of our code-as-contract approach.

"And there’s no separate licensing?"

One common source of pain for users of on-prem infrastructure has been license management: financial pain due to over-paying and under-utilizing, and operational pain in the navigation of different license terms, different expiration dates, unpredictable dependencies, and uncertain vendor futures. From the beginning we knew that we wanted to deliver a delightful, integrated experience: we believe that cloud computers should come complete with all system software built-in, and with no additional licensing to manage or to pay for. Bug fixes and new features are always only an update away and do not require a multi-departmental discussion to determine value and budget.

"It’s all open source?"

While the software is an essential part of the Oxide cloud computer, what we sell is in fact the computer. As a champion of open source, this allows Oxide a particularly straightforward open source strategy: our software is all open. So you don’t need to worry about hinky open core models or relicensing surprises. And from a user perspective, you are assured levels of transparency that you don’t get in the public cloud — let alone the proprietary on-prem world.

Getting your own first impression

We’re really excited to have the first commercial cloud computer — and for it to be generally available! If you yourself are interested, we look forward to it making its first impression on you — reach out to us!