Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experiment · HackerPapers discussions

Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experiment — mentioned by godelski

Mon, 17 Jul 2023 18:55:19 GMT

> paper reviewers are usually not supposed to look at the actual source code of the papers Wait what? I haven't reviewed for ACL but most conferences don't say "don't look at the source code." They will say that reviewers are not required to look at it (as well as the appendix). But generally it just isn't uploaded. I do always look at the main method when it is there but talking to my peers and advisor, this is very uncommon[0]. My experience is that most reviewers do not spend more than an hour on a work and make an opinion within 15 minutes. > Not sure what the best solution is, other than…

Paper and discussion history. A mention is not an endorsement.

Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experiment — mentioned by godelski

Fri, 05 May 2023 21:09:35 GMT

Honestly, these complaints (other than 4) apply to the vast majority of papers. #4 is just false. It has already been viewed by other lab members (peers) and open publication is peer reviewing. The "peer review system" (publishing to conferences/journals) is relatively new and I think ML demonstrates all the problems with the system (yay hype). Novelty is especially a joke. ViTs are "just" NLP encoding transformers. T2I models are "just" NLP models connected to generative models. Diffusion models are "just" whitening models. GPT3 is just GPT2 with more layers and more data which is just GPT…

Paper and discussion history. A mention is not an endorsement.

Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experiment — mentioned by godelski

Mon, 06 Mar 2023 19:17:42 GMT

So there's a few things to consider: First, this peer review via conferences/journals/etc is relatively new in the scientific process. Really only the last 50 years has this paradigm been the main way for publishing. Prior to that scientists have just published in the open and and peer review happened by peers reading and responding. Not too different from what we see with arxiv, twitter, and blogging. Second, we need to talk about how good the review process actually is. There's been a lot of writing on the NeurIPS experiments [0] is the most famous one. But the Google paper[1] notes that…

Paper and discussion history. A mention is not an endorsement.

Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experiment — mentioned by moyix

Tue, 05 Jul 2022 19:14:07 GMT

cperciva, if you're around, I'd be really curious to hear a bit more about the details behind your attempts to get "Cache Missing for Fun and Profit". I'm really surprised that it was so hard to find a home for it! Maybe a more general security venue like USENIX Sec, ACM CCS, or IEEE S&P would have been a better fit for something that was neither fish nor fowl? Just a couple years later in 2008 USENIX Sec published "Reverse-Engineering a Cryptographic RFID Tag" which is a personal favorite and similarly is a mix of low-level hardware architecture and cryptography. There were two other shared…

Paper and discussion history. A mention is not an endorsement.

Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experiment — HN submission

Tue, 28 Sep 2021 15:47:40 GMT

Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experiment

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experiment — HN submission

Wed, 22 Sep 2021 12:15:06 GMT

Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experiment

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.