There has been recent talk about a consistent finding of scientific studies initially overstating effects, which become weaker and weaker with additional scrutiny. This to me is an artifact of both the difficulty of designing a truly unbiased experiment and the publication process. Mind you, there’s no foul play at work here; science is just a little less “free to explore” than most people believe it to be. But the insistence that results always be “good” to be worthwhile ultimately harms its objectivity.
Statistical methods have become very good at reducing variance, and most researchers make a large concern out of achieving adequate sample sizes and creating complex statistical models in an attempt not only to demonstrate an effect, but to demonstrate that it cannot possibly be due to chance (more accurately, we think it’s not more than 5% likely, anyway :))
Unfortunately, while variance is usually fairly easy to detect and deal with, there is another contributor to statistical error: bias. Each experimental setup introduces its own bias. If we assume that the biases of independent experiments are roughly random (i.e. the biases are unbiased :)), then we would expect a possible over or understatement of an effect in the beginning, with a gradual regression to its true prevalence as additional models are “averaged into” the literature.
But the biases which have been observed are positive only. Effects are found to be *stronger* initially rather than weaker.
Here is where publication bias enters the scene: it is acceptable to publish *new* work only if the results of that work are “good”, whatever that may mean in a given field, and experimental parameters and models will be adjusted until such results are achieved (resulting in more than a few cases of statistical overfitting, I’m sure). Negative results are not published; they are either improved or abandoned.
But while this is true for pioneering work, it is NOT true for subsequent “reviews” of said work, which can be published simply on the basis of picking apart an existing effect.
This virtually guarantees that the initial bias will be an overstatement, and the subsequent direction with more study will push it in the negative direction, presumably until – eventually – the true effect is approximated.