Should KubeCon be double-blind?

With a paltry 13% acceptance rate, KubeCon is naturally going to generate a lot of disappointment – the vast, vast majority of proposals aren’t being accepted. But as several have noted, a small number of vendors account for a significant number of accepted talks. Is this an issue? In particular, review for KubeCon isn’t double-blind; should it be?

In terms of my own perspective here, I view conferences for practitioners (and especially their concomitant hallway tracks) as essential for the community of our craft. Historically, I have been troubled by the strangulation of practioner conferences by academic computer science: after we presented DTrace at USENIX 2004, I publicly wondered about the fate of USENIX – which engendered some thoughtful discussion. When USENIX had me keynote their annual technical conference twelve years later, I used the opportunity to express my concerns with the conference model, and wondered about finding the right solution both for practitioners and for academic computer science. That evening, we had a birds-of-a-feather session, which (encouragingly) was very well attended. There were many interesting perspectives, but the one that stood out to me was from Kathryn McKinley, who makes a compelling case that reviews should be double-blind. In the BOF, McKinley was emphatic and persuasive that conferences absolutely must be double-blind in their review – and that anything less is a disservice to the community and the discipline.

Wanting to take that advice, when we organized Systems We Love later that year, we ran it double-blind with a very large (and, if I may say, absolutely awesome!) program committee. We had many, many submissions – well over ten times the number of slots! We were double-blind for the first few stages of review, until the number of submissions had been reduced by a factor of five. Once we had reduced the number of talks submissions to “merely” double the number of slots, we de-blinded to get the rest of the way to a program. (Which was agonizing – too many great submissions!) By de-blinding, we were essentially using factors about the submitter as a tie-breaker to differentiate submissions that were both high quality – and as a way to get voices we might not otherwise hear from.

Personally, I feel that we were able to hit a sweet spot by doing in this way – and there were quite a few surprises when we de-blinded. Of note, at least a quarter of the speakers (and perhaps more, as I didn’t ask everyone) were presenting for the first time. Equally as surprising: several “big names” had submissions that we rejected while blinded – but looking at their submissions, the submissions themselves just weren’t that great! (Which isn’t to say that they don’t have a ton of terrific work to their name – just that every swing of the bat is not going to be a home run.)

So: should KubeCon be double-blind? I consider myself firmly in McKinley’s camp in that I believe that any oversubscribed conference needs to be double-blind to a very significant degree. That said, I also think our challenges as practitioners don’t exactly map to the challenges in academic computer science. (For example, because we aren’t using conferences as a publishing vector, I don’t think we need to be double-blind-until-accept – I think we can de-blind ourself to our rejections.) I also don’t even think we need to be double-blind all the way through the process: we should be double-blind until the program committee has reduced the number of submissions to the point that every remaining submission is deemed one that the program committee wants to accept. (That is, to the point that were it not for the physical limits of the conference, the program committee would want to accept the remaining submissions.) De-blinding at this point assures that the quality of the content is primarily due to the merit of the submission – not due to the particulars of the submitter. (That is, not based on what they’ve done in the past – or who their employer happens to be.) That said, de-blinding at the point of quality does allow these other factors to be used to mold the final program.

For KubeCon – and for other practitioner conferences – I think a hybrid model is the best approach: double-blind for a significant fraction of review, de-blinded for a final program formulation, and then perhaps “invited talks” for talks that were rejected when blind, but that the program committee wishes to accept based on the presenter. This won’t lead to less disappointment at KubeCon (13% is too low an acceptance rate to not be rejecting high-quality submissions), but I believe that a significantly double-blind process will give the community the assurance of a program that best represents it!