The contagion of fear

I have a difficult confession to make — a shameful incident that I have never spoken of, even to those closest to me.

Late one night in my first year at university and having found computer science to be my life’s calling, I was in the back of the large computer science lab, talking with some fellow underclassmen. We were snidely decrying the lack of technological understanding in the broader population, with the kind of arrogance and hubris that youth and precociousness can uniquely summon. Somewhere in this discussion, a terrible idea was hatched: what if we went next door to the general computer lab, and told the humanities students there that a computer virus was spreading through their machines? Would they believe it?

To my everlasting regret, we acted on this ugly urge: we went next door to a packed lab of students working on term papers, and — with synthetic alarm in our voices — loudly announced that a virus had escaped the computer science lab, and that they all needed to eject their floppy disks to prevent its spread.

What happened next was bedlam — the technological equivalent of shouting fire in a crowded theater. People panicked: they not only ejected floppies, but they powered machines off; yanked cables out of already-powered off machines; screamed; ran out of the room. We immediately realized that we had done something awful, but found that we couldn’t contain the wildfire that we had lit. We tried to explain that it was "a prank"; some (rightfully!) responded with indignant fury (what was wrong with us that we would do such a thing?!) while others simply didn’t believe us: once the fear had taken root, we couldn’t eradicate it.

The aftermath is a blur — but it was bad. It was the week leading up to finals, and people who had powered their computers off had lost work. They were furious, and we were in a ton of trouble.

The next morning, we found ourselves in the office of the facilities director (a famously imposing presence), who lectured us sternly as we stared at our feet. He made us each write letters of apology (easily done as my shame was earnest), telling us to give him two copies: one to be sent to those whom we had wronged, the other to be kept in his drawer. He told us in no uncertain terms that if we ever found ourselves anything close to anything like this again, we would be finding another university to attend — and we assured him that we had learned our lesson. (When I graduated four years later, he made a point of handing the letter back to me — and I remain grateful for the gravity and empathy with which he handled the whole thing.)

I could have (and perhaps would have?) taken this whole shameful incident to the grave, but I feel compelled to confess it now, because I have never seen fear sown so irresponsibly by putative technologists as I have now with respect to AI and the threat of human extinction (!). Amazingly, that is not hyperbole: this week, ex-Anthropic employee Jacob Coxon claimed — and then Anthropic Alignment Science lead Evan Hubinger agreed — that the probability that AI will "kill all humans" is ">10% in the next decade" (!!).

That is, the claim is not merely that AI could kill thousands, millions, or even billions of humans (claims that themselves would be extraordinary!), but that there is a greater than ten percent chance that AI will kill all humans. That is, if you have a newborn, their claim is that there is a greater than ten percent chance that your child will somehow perish at the hands of AI before they enter middle school.

These ghoulish claims strike brazenly at the hearth, and given the obvious importance of AI, it is unsurprising that they have leapt into the mainstream, with people asking the natural question: how would that happen? The answers always rely on hand-wavy extrapolation into the future; for example, Jacob Coxon cites "hacking critical infrastructure" and "extinction-level bioweapons" without further elaboration. But Coxon is not an expert on critical infrastructure, nor on bioweapons — nor, for that matter, on extinction.

What 27-year-old Coxon is an expert on — if accidentally — is the contagion of fear. We cannot expect the public to understand the inner workings of all technology (that was the fallacy of my 18-year-old self!): technology is too broad, too deep, and too core to all of our lives to expect everyone to understand everything; we must — at some level — trust domain experts. When those domain experts sow fear, it is the fear that takes root, propagating much more readily than any evidence that may follow it. (As Jonathan Swift famously quipped over three centuries ago, "Falsehood flies, and the truth comes limping after it".) In this regard, Coxon is more vector than index case: it is indisputable that the fears of extinction risk are widespread and Coxon himself likely has these fears because he has heard them from someone else. Unfortunately, as the fear takes hold, the sheer number of frightened experts becomes its own kind of evidence, with contrary voices drowned out by the din of an apparent consensus.

With the full knowledge that the fear is becoming well established, I would nonetheless like to offer a rebuttal, if from my own (narrow) domain of expertise: building computers. As I reflected in my 2023 Monktoberfest talk, Intelligence is not Enough, acts of engineering are not acts of intelligence alone. They require not only the human aspects of our character, they also require our action in (and reasoning about) the physical world. Those who are resolute in their fear like to gloss over these bits ("robots will do this!"), but to do so is to ignore the physical reality that robots cannot do this today — and do not seem to be doing so on a time horizon consistent with these extinction fears.

More generally, to the degree that digital systems have agency in the physical world, it is agency that we — we humans with arms and legs and brains and parents and children — permit: AI executes on physical systems that have been engineered with human accountability and control. Intelligence does not exempt a system from the realities of the physical world!

To those who — like Coxon — are gripped with unwavering fear, this will be of little solace. But to those who are hearing these fears for the first time: know that your incredulity is warranted. We should heed the wisdom of the late Carl Sagan that extraordinary claims require extraordinary evidence: the claims from Coxon and his ilk are the most extraordinary a technologist can make, and we must demand evidence commensurate with the claims.

That said, we should not expect the public to understand LLMs, critical infrastructure, bioweapons, extinction biology, etc. — that burden must lie with those making the claim. The lesson that I learned (shamefully) decades ago is that domain experts, by way of their expertise, implicitly hold the public’s trust — and we must not abuse it. It is incumbent upon us to be circumspect in our claims — and maximally so when raising the alarm.

So to those in that late-night lab over three decades ago, I’m sorry: you had nothing to fear then — and (at least with respect to extinction risk!) you have nothing to fear now.