Holmwood & McKay on the sociology REF

John Holmwood and Stephen McKay have a very interesting piece on sociology and the REF at the THES blog.  Among other things they point out that certain sub-panels, for instance sociology and anthropology and development studies gave less widely dispersed scores than others, for instance economics and econometrics and social work and social policy. My immediate thought is  that this is exactly what you would expect. Why so? Well, one feature that sociology, anthropology and development studies probably share is a lack of  consensus as to what is to count as good work in the field ie they are disciplines of low disciplinarity. 

I'm now going to make a conjecture. Let's assume that a consequence of "standards ambiguity" is that a randomly selected pair of assessors in  the sociology sub-panel are likely to produce a larger difference in their ratings of a randomly selected sociology REF submitted article than a randomly selected pair of assessors from the economics sub-panel rating would of a randomly selected economics article.  In other words there is more agreement in economics about what is good and bad. Now assume that the discrepancies between the raters are resolved by averaging the two ratings. The effect  will be to pull scores into the middle of the distribution to  greater extent in the sociology than in the economics sub-panel.

We all in fact can think of another example of this phenomena - the "double marking" of student essays, where, mysteriously these days practically everyone ends up with a 2.1 and everybody scratches their head and says: "I wonder how that happened?".

I've no idea whether discrepancies between sub-panel reader's ratings were indeed reconciled by an averaging process. But it is likely that something of the sort went on, even if it was disguised as having a discussion to reconcile differences. Again experience of examining suggests to me that the latter always tends towards a "let's split the difference" solution.

Signals of quality are clear in economics. Everyone knows what the good journals are and gives weight to publication in them. This is not true in sociology where it is perfectly possible to claim,  that a paper in the American Sociological Review is only a 2* publication while a publication in Body & Society is 4* and still be taken seriously by fellow REF panel members.

An apparent puzzle is why there is more agreement in social policy? But maybe we shouldn't be so puzzled. After all the policy relevance of a piece should have some objective foundation and there are certain fields - for example evidence based policy - where there are agreed protocols for what counts as good work. It is probably also worth remembering that social policy is also a hiding place for a large number of decent applied economists, who presumably bring with them disciplinary expectations about standards.

One final thought. If the next REF takes anything like its present form and the sociology UOA is abolished, one possibility is that sociology will be merged with social policy and social work. If that is the case look out for a great equalization of scores and, presumably, a massive increase in the number of sub-panel members to cope with the increased diversity of intellectual projects they will have to reach a judgement about.