Think Hard, Build Harder

Summary

Overall, this article provides a holistic insight into how published academic studies are likely to be flawed, why, and why replication is not going to be easy.

Nuggets

• Type I error: thinking something is true when it is not (false positive); type II error: thinking something is not true when in fact it is (false negative);
• Statistical power of the study: a measure of a study’s ability to avoid type II errors (a real signal is missed as noise). For example, a power of 0.8 means that on average only two out of ten true hypothesis will be misidentified as false.
• “According to some estimates, three quarters of published scientific papers in the field of machine learning are bunk because of this ‘overfitting’, says Sandy Pentland, a computer scientist at the Massachusetts Institute of Technology”

Take-Away

For HCI research, before we start a new wave of criticisms on our published studies, let’s take a minute and ask ourselves: actually, how important is study to the research of HCI?

The HCI papers that influence me the most are never known for their studies, e.g., Tangible Bits, Sensing techniques for mobile interactions, NaviCam. Looking at these papers, their impact is really NOT about whether there was any p smaller than .05 or any results that broke the ground. Always, it’s the idea, the vision, the realization that computer can be made like this that got people excited, and subsequently, inspired a generation of exploration that made  interactive computing what it is today. Study is admittedly important for HCI in general, but in my opinion it shouldn’t be something research folks care too much. I think for HCI, designing a study right, making a wrong study correct or replicating a study to overturn its result is indeed worthy, but doesn’t necessarily warrant a significant contribution that truly propel the advance of the field.