Popular Posts

Caveat Emptor

The opinions expressed on this page are mine alone. Any similarities to the views of my employer are completely coincidental.

Friday 26 September 2014

This paper is so bad it is not even wrong

For a long time I've been toying with the idea of  post publication review. It seems to me that, at least in British sociology, peer review is so badly broken that  though nuggets of gold undoubtedly exist, the dung heap of illogicality, nonsense and  muddleheadedness is so huge that they can be difficult to find. The consequence is that zombie research lives on, is among us, and frequently gets cited, often by people who should know better and probably would know better if they actually bothered to read and think about the things they cite.

But how to do it? One idea I have is from time to time to take an issue of a journal, read all the articles in it, and  for each  one post a  paragraph  precis in plain unadorned English somewhat in the spirit of Jim Crace's pomposity busting Digested Read. My intuition is that many, when treated in this way, will, sans bullshit,  turn out to be assertions  of the "bleeding obvious" or just incoherent.

Of course this won't cover all bases. It's possible to have a perfectly coherent question but  treat it in an inadequate or incompetent way and by doing so end up with the wrong answer. Dealing with that sort of thing requires more space  than a single paragraph. It also requires more patience from the reader. 

I'm convinced that the reason so much bad quantitative work is published in British sociology journals is that there are so few consumers whose critical faculties extend much beyond name checking  whether a particular recipe has been used. I usually call this the "That's not the way we learned to do it in grad school" syndrome. Reaching that sort of reader is hard, because it assumes the ability to follow a chain of reasoning which is a little more subtle than: it's a binary response so therefore a logistic regression is correct.

I'm still at the stage of wondering whether any of this is a good idea or a real dog. One thing is clear, it wouldn't make me many friends; well, I'm used to that, but even I recognize that there are psychological limits to how many battles can be fought simultaneously.

But if I do go ahead I'll have to think carefully about a catchy title for the series. At the moment I have three contenders,  all from  Wolfgang Pauli:

This paper is so bad it is not even wrong.

I don't mind you thinking slowly; I mind your publishing faster than you think.

The setup...as far as printing and paper are concerned is splendid.


No comments: