For a long time I've been toying with the idea of post publication review. It seems to me that, at least in British sociology, peer review is so badly broken that though nuggets of gold undoubtedly exist, the dung heap of illogicality, nonsense and muddleheadedness is so huge that they can be difficult to find. The consequence is that zombie research lives on, is among us, and frequently gets cited, often by people who should know better and probably would know better if they actually bothered to read and think about the things they cite.
But how to do it? One idea I have is from time to time to take an issue of a journal, read all the articles in it, and for each one post a paragraph precis in plain unadorned English somewhat in the spirit of Jim Crace's pomposity busting Digested Read. My intuition is that many, when treated in this way, will, sans bullshit, turn out to be assertions of the "bleeding obvious" or just incoherent.
Of course this won't cover all bases. It's possible to have a perfectly coherent question but treat it in an inadequate or incompetent way and by doing so end up with the wrong answer. Dealing with that sort of thing requires more space than a single paragraph. It also requires more patience from the reader.
I'm convinced that the reason so much bad quantitative work is published in British sociology journals is that there are so few consumers whose critical faculties extend much beyond name checking whether a particular recipe has been used. I usually call this the "That's not the way we learned to do it in grad school" syndrome. Reaching that sort of reader is hard, because it assumes the ability to follow a chain of reasoning which is a little more subtle than: it's a binary response so therefore a logistic regression is correct.
I'm still at the stage of wondering whether any of this is a good idea or a real dog. One thing is clear, it wouldn't make me many friends; well, I'm used to that, but even I recognize that there are psychological limits to how many battles can be fought simultaneously.
But if I do go ahead I'll have to think carefully about a catchy title for the series. At the moment I have three contenders, all from Wolfgang Pauli:
This paper is so bad it is not even wrong.
I don't mind you thinking slowly; I mind your publishing faster than you think.
The setup...as far as printing and paper are concerned is splendid.
No comments:
Post a Comment