Preserving Statistical Validity in Adaptive Data Analysis · HackerPapers discussions

Preserving Statistical Validity in Adaptive Data Analysis — mentioned by majos

Tue, 29 Oct 2019 16:34:26 GMT

It's worth distinguishing between different types of bad science. 1. The first and most egregious type is outright fraud. This is when you intentionally manipulate or fake data. Everyone agrees this is bad, and honest actors are enough to prevent it. In some cases, other honest actors are sufficient to determine if the claims are fishy. 2. The second more subtle type is not paying attention to adaptivity. For example, maybe an investigator wants to look at the data before coming up with a hypothesis to test. This is dangerous because the investigator is already overfitting, so any p-values…

Paper and discussion history. A mention is not an endorsement.

Preserving Statistical Validity in Adaptive Data Analysis — mentioned by majos

Sun, 28 Jan 2018 02:06:15 GMT

This article makes good points. For me the most salient is that the current reward structure ("how many strange new positive results have you published?") skews published research across almost all scientific fields, and that science in general needs to find a better way to incentivize the unglamorous work of reproducing splashy results and useful failures. The suggested method of pre-registering hypotheses is certainly one way to address these issues. There are also more complicated methods of attacking self-fulfilling analyses through clever statistical tools [1], although I think this…

Paper and discussion history. A mention is not an endorsement.