I don’t quite get this article by Robert Gebelhoff at the Washington Post.

Sure, there are well-known pressures on academics to publish significant results, and also to get media attention. But those are conceptually distinct issues. Publication bias (and related problems like p-hacking and the Garden of Forking Paths) tend to inflate the statistical significance of published results. But that’s not related to the substantive significance of the results, to how interesting the questions being answered are. Solving publication bias would not stop journals privileging articles about exciting or controversial topics. Why would you even want that?

More than that, why reserve specific criticism for Stasko and Geller’s paper? Gebelhoff says that

the research had not been published in any academic journal. Instead, the data was compiled through an Internet survey as part of a presentation to the American Psychological Association’s annual convention. Sure, the results were interesting, but the research is simply not generalisable to the entire public.

That doesn’t quite scan either. The fact that the research wasn’t (yet?) published in a journal has got nothing to do with the fact that they recruited candidates online. Journals publish online studies all the time. This paper was accepted to and presented at a conference, which generally implies at least some level of filtering by peers, even if it’s short of peer-review. Peer-review isn’t a guarantee of accuracy either.

More than that, would you lend more credence to a typical published psychology study where the sample wasn’t Internet randos but university students recruited from a Psych 101 class? Because the results you get from studying undergrads people don’t generalise well either.¹ If anything, I’d be more convinced (ceteris paribus) by an attitudinal study about new technologies where the sample age range was 18–82! This paper isn’t primarily making claims about the prevalence of behaviours in the population, but about relationships between them. So the fact that it isn’t a random sample from the population is important but not critical.

There are a number of issues with the ways science and the media operate. But even when they affect each other, they remain distinct issues.