The Observation Deck

Close this search box.

Should KubeCon be double-blind?

October 3, 2018

With a paltry 13% acceptance rate, KubeCon is naturally going to generate a lot of disappointment — the vast, vast majority of proposals aren’t being accepted. But as several have noted, a small number of vendors account for a significant number of accepted talks. Is this an issue? In particular, review for KubeCon isn’t double-blind; should it be?

In terms of my own perspective here, I view conferences for practitioners (and especially their concomitant hallway tracks) as essential for the community of our craft. Historically, I have been troubled by the strangulation of practioner conferences by academic computer science: after we presented DTrace at USENIX 2004, I publicly wondered about the fate of USENIX — which engendered some thoughtful discussion. When USENIX had me keynote their annual technical conference twelve years later, I used the opportunity to express my concerns with the conference model, and wondered about finding the right solution both for practitioners and for academic computer science. That evening, we had a birds-of-a-feather session, which (encouragingly) was very well attended. There were many interesting perspectives, but the one that stood out to me was from Kathryn McKinley, who makes a compelling case that reviews should be double-blind. In the BOF, McKinley was emphatic and persuasive that conferences absolutely must be double-blind in their review — and that anything less is a disservice to the community and the discipline.

Wanting to take that advice, when we organized Systems We Love later that year, we ran it double-blind with a very large (and, if I may say, absolutely awesome!) program committee. We had many, many submissions — well over ten times the number of slots! We were double-blind for the first few stages of review, until the number of submissions had been reduced by a factor of five. Once we had reduced the number of talks submissions to “merely” double the number of slots, we de-blinded to get the rest of the way to a program. (Which was agonizing — too many great submissions!) By de-blinding, we were essentially using factors about the submitter as a tie-breaker to differentiate submissions that were both high quality — and as a way to get voices we might not otherwise hear from.

Personally, I feel that we were able to hit a sweet spot by doing in this way — and there were quite a few surprises when we de-blinded. Of note, at least a quarter of the speakers (and perhaps more, as I didn’t ask everyone) were presenting for the first time. Equally as surprising: several “big names” had submissions that we rejected while blinded — but looking at their submissions, the submissions themselves just weren’t that great! (Which isn’t to say that they don’t have a ton of terrific work to their name — just that every swing of the bat is not going to be a home run.)

So: should KubeCon be double-blind? I consider myself firmly in McKinley’s camp in that I believe that any oversubscribed conference needs to be double-blind to a very significant degree. That said, I also think our challenges as practitioners don’t exactly map to the challenges in academic computer science. (For example, because we aren’t using conferences as a publishing vector, I don’t think we need to be double-blind-until-accept — I think we can de-blind ourself to our rejections.) I also don’t even think we need to be double-blind all the way through the process: we should be double-blind until the program committee has reduced the number of submissions to the point that every remaining submission is deemed one that the program committee wants to accept. (That is, to the point that were it not for the physical limits of the conference, the program committee would want to accept the remaining submissions.) De-blinding at this point assures that the quality of the content is primarily due to the merit of the submission — not due to the particulars of the submitter. (That is, not based on what they’ve done in the past — or who their employer happens to be.) That said, de-blinding at the point of quality does allow these other factors to be used to mold the final program.

For KubeCon — and for other practitioner conferences — I think a hybrid model is the best approach: double-blind for a significant fraction of review, de-blinded for a final program formulation, and then perhaps “invited talks” for talks that were rejected when blind, but that the program committee wishes to accept based on the presenter. This won’t lead to less disappointment at KubeCon (13% is too low an acceptance rate to not be rejecting high-quality submissions), but I believe that a significantly double-blind process will give the community the assurance of a program that best represents it!

Leave a Reply

Recent Posts

November 18, 2023
November 27, 2022
October 11, 2020
July 31, 2019
December 16, 2018
September 18, 2018
December 21, 2016
September 30, 2016
September 26, 2016
September 13, 2016
July 29, 2016
December 17, 2015
September 16, 2015
January 6, 2015
November 10, 2013
September 3, 2013
June 7, 2012
September 15, 2011
August 15, 2011
March 9, 2011
September 24, 2010
August 11, 2010
July 30, 2010
July 25, 2010
March 10, 2010
November 26, 2009
February 19, 2009
February 2, 2009
November 10, 2008
November 3, 2008
September 3, 2008
July 18, 2008
June 30, 2008
May 31, 2008
March 16, 2008
December 18, 2007
December 5, 2007
November 11, 2007
November 8, 2007
September 6, 2007
August 21, 2007
August 2, 2007
July 11, 2007
May 20, 2007
March 19, 2007
October 12, 2006
August 17, 2006
August 7, 2006
May 1, 2006
December 13, 2005
November 16, 2005
September 13, 2005
September 9, 2005
August 21, 2005
August 16, 2005