Holmwood & McKay on the sociology REF

John Holmwood and Stephen McKay have a very interesting piece on sociology and the REF at the THES blog.  Among other things they point out that certain sub-panels, for instance sociology and anthropology and development studies gave less widely dispersed scores than others, for instance economics and econometrics and social work and social policy. My immediate thought is  that this is exactly what you would expect. Why so? Well, one feature that sociology, anthropology and development studies probably share is a lack of  consensus as to what is to count as good work in the field ie they are disciplines of low disciplinarity. 

I'm now going to make a conjecture. Let's assume that a consequence of "standards ambiguity" is that a randomly selected pair of assessors in  the sociology sub-panel are likely to produce a larger difference in their ratings of a randomly selected sociology REF submitted article than a randomly selected pair of assessors from the economics sub-panel rating would of a randomly selected economics article.  In other words there is more agreement in economics about what is good and bad. Now assume that the discrepancies between the raters are resolved by averaging the two ratings. The effect  will be to pull scores into the middle of the distribution to  greater extent in the sociology than in the economics sub-panel.

We all in fact can think of another example of this phenomena - the "double marking" of student essays, where, mysteriously these days practically everyone ends up with a 2.1 and everybody scratches their head and says: "I wonder how that happened?".

I've no idea whether discrepancies between sub-panel reader's ratings were indeed reconciled by an averaging process. But it is likely that something of the sort went on, even if it was disguised as having a discussion to reconcile differences. Again experience of examining suggests to me that the latter always tends towards a "let's split the difference" solution.

Signals of quality are clear in economics. Everyone knows what the good journals are and gives weight to publication in them. This is not true in sociology where it is perfectly possible to claim,  that a paper in the American Sociological Review is only a 2* publication while a publication in Body & Society is 4* and still be taken seriously by fellow REF panel members.

An apparent puzzle is why there is more agreement in social policy? But maybe we shouldn't be so puzzled. After all the policy relevance of a piece should have some objective foundation and there are certain fields - for example evidence based policy - where there are agreed protocols for what counts as good work. It is probably also worth remembering that social policy is also a hiding place for a large number of decent applied economists, who presumably bring with them disciplinary expectations about standards.

One final thought. If the next REF takes anything like its present form and the sociology UOA is abolished, one possibility is that sociology will be merged with social policy and social work. If that is the case look out for a great equalization of scores and, presumably, a massive increase in the number of sub-panel members to cope with the increased diversity of intellectual projects they will have to reach a judgement about.

How anti-semitic are you?

So, I'm sitting at the breakfast table this morning trying to jolly along my eight year old to finish her breakfast so we can start the school run and in the background, as usual, is Radio 4. It's news time and one of the top 3 items is a  story on anti-semitism that  claims that nearly 50% of the British population has at least some anti-semitic views and that 50% of British Jews now believe they have no long term future in Britain. This is reported in a pretty matter of fact way, with no comment and no attribution (that I can recall) . 

To me these numbers seem completely unbelievable, but maybe I just live in a liberal-lefty bubble in which I only ever meet like minded people who just confirm my rosy view of the world. Luckily I do have some horse sense and at least a dim awareness of the informational biases that we are all subject to.

I have no difficulty in believing that there exists in Britain a number of people with an unhealthy interest in uniforms, armbands and shiny black boots. Some of them may even have a suspicious interest in sepia tinted photographs of a man with a very odd mustache. Their grasp of the contours of modern European history may also, in some respects, be a bit shaky. A lot of them have, and if asked will express, obnoxious views about Jews, Muslims, Irishmen, homosexuals, Asians, Africans, socialists, liberals and just about any other minority group you can name. Let's face it, the nutters are always with us, sometimes a few more, sometimes a few less.

But 50% of the British population is in some sense anti-semitic? If that's true then I really am living in Cloud Cuckoo Land and I should be paying more attention. With that thought in mind I decided to track down the sources. First stop after reaching work is to sit down with a coffee and pick up the Guardian. Nothing in the print version. I do better with the Times and discover one column on an inside page. This gives me the information that YouGov are responsible for some of the data the claims are based on.

So over to my office and onto the internet. First stop YouGov's website. No information there. OK, time for a general Google trawl. Eureka! Most of the British qualities and a few of the tabloids have a report in their online versions. It turns out that the data claims are based on a report put out by a group called Campaign Against Antisemitism. OK, now we are in business. CAA are a lobby and advocacy group. That's fine, nothing wrong with advocacy research as long as everyone is clear about what it is.

It's easy to find out who is fronting CAA though not where its funds come from. It was set up in August 2014 around the time the last Israel-Gaza conflict kicked off and was involved in the campaign to get the Tricycle Theatre to accept the London Israeli Embassy's sponsorship of the UK Jewish Film Festival. Just to get the facts straight the Tricycle, which had hosted the UKJFF for a number of years, wished to carry on doing so, but did not wish to accept sponsorship by the Israeli Embassy and offered to make up the shortfall from its own funds. This was regarded by some as evidence of anti-semitism on the part of the Tricycle. Jonathan Sacerdoti, CAA's Director of Communications is a well known lobbyist who appears to have a certain amount of form including some pretty crude attempts to cover up his tracks and present himself as a disinterested commentator on Middle-Eastern affairs.

So to sum up the story so far, the BBC were reporting the "findings" of an advocacy group closely linked to Britain's pro Israel lobby. Just to be clear, I'm not taking a view one way or other as to the rights or wrongs of this: people are free to push whatever they think they can sell to the media. I would have appreciated being told what the source was though so that I could take a more informed view as to whether I should believe it.

Now on to the "facts" about anti-semitism in Britain.

The evidence comes from a YouGov internet survey carried out on behalf of CAA. You can find the report here. Let's skip over all the generalized scepticism about  internet surveys, weighting, non-response and the rest and cut to the chase. The respondents were asked to rate whether seven statements about Jews were definitely true, probably true and presumably some other categories that we're not told about, which were taken to be indicators of anti-semitic views. Let's also gloss over the validity of the statements and simply take them at face value.

Here are the statements followed by the percentage saying they thought it definitely or probably true and in brackets the percentage thinking it untrue:

Jews think they are better than other people. 17% (83%)

In business, Jews are not as honest as most people. 11% (89%)

I would be unhappy if a family member married a Jew. 10% (90%)

Jews have too much power in the media. 17% (83%)

Jews chase money more than other British people. 25% (75%)

Jews' loyalty to Israel makes them less loyal to Britain than other British people. 20% (80%)

Jews talk about the Holocaust too much in order to get sympathy. 13% (87%)

So in fact, in all but one case 4 out of 5 Brits do not believe these statements to be true and 9 out of 10 would be happy for a family member to marry a Jew. Suddenly the British anti-semites seem quite a bit thinner on the ground. 

The headline "almost half (45%) of British adults believe at least one of the antisemitic statements shown to them" while no doubt literally true of those surveyed is not a sensible measure of the propensity towards anti-semitism in the British population. It has been produced by a kind of "Texan sharpshooter" procedure in which the target is made progressively larger until you include the required proportion of the population you are aiming at. 

If you are going to proceed in this way you need to know something about the base rate propensity for people to affirm  nonsensical statements or  statements about things about which they have no knowledge whatsoever.  In other words a fair proportion of the 45% that affirmed at least one of these statements could quite plausibly be counted an anti-semite purely as a result of a "tremblingly-hand" process. One might also wonder what the rate of affirmation would be to some of these statements if we substituted for "Jews" descriptors like "Christians", "Moslems" "Methodists" "Cockneys". Is someone who has a low opinion of everyone ipso facto an anti-semite?

Turning to the CAA survey of  British Jews, this is again an internet survey carried out over 20 days between 23rd December and 11th January. I think we can concede that this is a difficult population to survey while still holding that the representativeness of those surveyed is almost completely unknown. What is plausible from the description of the recruitment method given by CAA is that there is probably a bias towards the observant and against the secular and towards those who have or think they have experienced anti-semitism. 

There is probably also a bias towards those that view criticism of Israel as prima facie evidence of anti-semitism:

Boycotts of businesses selling Israeli products constitute intimidation. (84% agree)

Media bias against Israel fuels persecution of Jews in Britain. (82% agree)

I have witnessed antisemitism that was disguised as a political comment about Israel. (77% agree)

At least I hope that is the correct interpretation of these figures.  If it isn't then we really do have to wonder about what kind of a bubble some of us are living inside and what organizations like CAA are doing to inflate it.

And finally we can ask, why is the BBC giving such weight to the rather tendentious "research" of a lobby group, moreover a lobby group whose Director of Communications has burned their fingers before? Surely we should be told...

Steve Fuller on sociology and the REF

Over at The Sociological Imagination Steve Fuller has an interesting piece on sociology and the REF. I could quibble about a few minor things, but I'm in broad agreement with the general message and in particular his final paragraph which I'll quote here:

The specific lesson with regard to sociology is that the lack of consensus on the acceptance of the overall REF results – highlighted by the BSA’s general statement to members – may indeed be a sign of sociology’s loss of clear disciplinary identity in this country. What critics of the REF results claim to be the ‘harshness’ of the REF panel’s judgements may reflect the highly variable understanding of what counts as ‘sociology’. Yes, it may really matter who’s on the panel because there is no common understanding of the field. If so, then on what grounds can the idea of sociology as a ‘discipline’ be maintained? No doubt ‘sociology’ is a great market attractor to get students into specific degree courses – and long may it flourish! But you don’t need dedicated sociology departments or, more to the point, research units to do this. Thus, the value of having sociology continue as a ‘discipline’ should focus sociologists’ minds, given that the neo-liberal order is not designed to respect, let alone protect, the value of disciplines as such.

"Successful" disciplines - in the social sciences the paradigm case is probably economics - define relatively tightly what kind of thing is inside and outside of the tent. Standards about what is and is not an acceptable question or solution to a question are shared and there is a consensus about the suite of acceptable research strategies - though not necessarily agreement about the applicability of a particular tool in each case. All this is not to say that people agree all the time about all the questions and the ways to answer them, but people do agree about the sort of thing they are doing  to the extent that they can meaningfully talk to each other. The Economic Journal does not publish poems.

Paradoxically a "discipline" in which "anything goes", is, of course, not literally a discipline in which everything goes. It is simply a discipline in which quite arbitrary decisions about academic value  get made that depend on whoever has grabbed or promoted themselves to positions of power and influence. Sometimes these decisions coincide with what is reasonable, sometimes they are absurd. How they look will very much depend on what flavour of "sociology" the observer happens to prefer. Personally I prefer p. values to poetry (as sociology) but nobody was asking me.

The President of the BSA Lynn Jamieson has posted a message on the BSA website  which contains the following sentences:

But whatever it looks like from where you stand, I hope we could agree that it is crucial that we, as sociologists, present a collaborative and positive front and continue to value each other’s research regardless of institutional affiliation. Division is not in the interests of the future of our discipline.

The sentiment, in the context, is understandable but is this what we really should be doing? Agreeing to disagree without making a serious effort to identify the sources of the difference is really not healthy for a discipline. What it means is that one thing gets said in public and another in private where the important decisions are really taken. It will be painful but in the long run better to cut out the hypocrisy and for everyone to say openly whatever they think is true about their own and other people's research. If you think that what I do is no good, muddle-headed or simply wrong, then I want to hear about it. Maybe you are right and if, by your arguments you persuade me, I'll change my mind. Or maybe you are talking out of your arse and then I want to be able to explain to you why I think that is the case. I can see no intellectual good coming out of pretending to value  research  I believe to be nonsense in order to protect a discipline that is so fragmented that it is merely a flag of convenience.

There are too many bunkers and dark hiding places in sociology where people just hang out with their mates and massage each other's egos. Most so called "debates" are nothing of the sort. In fact the last thing that many seem to want is genuine debate in the sense of extended argument and evidence being brought to bear on their beliefs. Witness the British Social Class Debate in the pages of Sociology: where exactly was the debate? How many of the contributions actually said anything of substance about the original article? How many of the arguments that were raised were actually addressed satisfactorily in the rejoinder?

And who are the guilty men and women? Well, not obviously the REF panel. Like Jamieson I think we have to believe that they were doing the best they could given the (absurd) rules of a game that probably a significant fraction of them didn't believe in. The real criminals, in my view, are the people that have led and been influential in running the major British sociological institutions. The BSA Presidents, the editors of the journals, the Professors in the major departments, in short the people who could have led but instead sat back and let a hundred flowers bloom  while saying things in private like: "I know that what x says is correct but I can't be seen to endorse that because it will upset y and lead to z and then w won't like me and therefore I won't be invited to v."

The specific lesson with regard to sociology is that the lack of consensus on the acceptance of the overall REF results – highlighted by the BSA’s general statement to members – may indeed be a sign of sociology’s loss of clear disciplinary identity in this country. What critics of the REF results claim to be the ‘harshness’ of the REF panel’s judgements may reflect the highly variable understanding of what counts as ‘sociology’. Yes, it may really matter who’s on the panel because there is no common understanding of the field. If so, then on what grounds can the idea of sociology as a ‘discipline’ be maintained? No doubt ‘sociology’ is a great market attractor to get students into specific degree courses – and long may it flourish! But you don’t need dedicated sociology departments or, more to the point, research units to do this. Thus, the value of having sociology continue as a ‘discipline’ should focus sociologists’ minds, given that the neo-liberal order is not designed to respect, let alone protect, the value of disciplines as such. - See more at: http://sociologicalimagination.org/archives/16661#sthash.aBP3OfNJ.dpuf

he specific lesson with regard to sociology is that the lack of consensus on the acceptance of the overall REF results – highlighted by the BSA’s general statement to members – may indeed be a sign of sociology’s loss of clear disciplinary identity in this country. What critics of the REF results claim to be the ‘harshness’ of the REF panel’s judgements may reflect the highly variable understanding of what counts as ‘sociology’. Yes, it may really matter who’s on the panel because there is no common understanding of the field. If so, then on what grounds can the idea of sociology as a ‘discipline’ be maintained? No doubt ‘sociology’ is a great market attractor to get students into specific degree courses – and long may it flourish! But you don’t need dedicated sociology departments or, more to the point, research units to do this. Thus, the value of having sociology continue as a ‘discipline’ should focus sociologists’ minds, given that the neo-liberal order is not designed to respect, let alone protect, the value of disciplines as such. - See more at: http://sociologicalimagination.org/archives/16661#sthash.aBP3OfNJ.dpuf

The specific lesson with regard to sociology is that the lack of consensus on the acceptance of the overall REF results – highlighted by the BSA’s general statement to members – may indeed be a sign of sociology’s loss of clear disciplinary identity in this country. What critics of the REF results claim to be the ‘harshness’ of the REF panel’s judgements may reflect the highly variable understanding of what counts as ‘sociology’. Yes, it may really matter who’s on the panel because there is no common understanding of the field. If so, then on what grounds can the idea of sociology as a ‘discipline’ be maintained? No doubt ‘sociology’ is a great market attractor to get students into specific degree courses – and long may it flourish! But you don’t need dedicated sociology departments or, more to the point, research units to do this. Thus, the value of having sociology continue as a ‘discipline’ should focus sociologists’ minds, given that the neo-liberal order is not designed to respect, let alone protect, the value of disciplines as such. - See more at: http://sociologicalimagination.org/archives/16661#sthash.aBP3OfNJ.dpuf