GBCS again...

Oh dear, I'm turning into the sort of obsessive bore that people cross to the other side of the room to avoid... but indulge me one more time. Michael Orwell, a producer at the BBC has posted an account of the Great British Class Calculator on the BBC's website. It would be unfair to blame Savage et al. for the (mis)representations of  the PR machine -  it's possible that they are not  even aware of them -  so I'm not going to direct my scorn at them.

Though it doesn't literally say it, a casual reading of  the piece conveys the impression that the 160,000 odd cases collected by the internet survey were used to construct the new class schema. This is, of course, totally incorrect. The BBC's internet survey had effectively no role whatsoever in defining the classes.

He (Orwell) then goes on to say:

"Using complex analytical statistics Mike and Fiona’s research team were able to place all 161,000 participants of the original BBC Lab UK survey into one of the new seven classes, robustly and with great accuracy" 

 The first part of this sentence is true, but there is no publicly available information that would allow anyone to evaluate the "robustness" and "accuracy" of the allocation procedure. It's not even clear what these impressive and confidence inspiring terms mean. 

The class categories were the result of a statistical analysis of roughly 1000 cases collected by face-to-face interviews with a quota sample of respondents.  The non probabilistic sampling method means that strictly speaking it is impossible to estimate sensible confidence intervals around quantities like the estimated proportion of the population allocated to each class, but common sense suggests that if you could they would be quite wide. 

No information is available about "robustness" - one interpretation of which would be the extent to which the composition of the class categories (in terms of the defining variables) changed when we assume the existence of say six or eight rather than seven classes.

And finally it is completely mysterious what "accuracy" means in the context of this sort of avowedly "inductive" study. If you have a gold standard classification you can see how a new method performs in allocating cases to it. This works because you know what the right answer is. But to talk about accuracy when you are making the classification up as you go along makes no sense whatsoever.

My conclusion is that "robustly" and "accuracy" are just farmyard noises that are meant to convey the impression of science to a public that is not in a position to know any better. And why does this matter? Well, it tells us something about the regard in which sociology is held. It seems that sociology is of so little consequence that it doesn't really matter how you describe sociological research. Anything that sounds plausible will do. I really don't know how the discipline can survive if that is the prevalent attitude.