I believe in genuinely open discussion, so when David Byrne contacted me about my post on his Sociology article on quantitative methods I offered him the opportunity to make a reply in a guest post. So here it is, a new departure in Oxford Sociology, the first guest post. I'll be responding in due course (you wouldn't, gentle reader, expect anything less).
David Byrne
Reply to Colin Mills
Colin Mills took exception to my article in
Sociology[i]
on a variety of grounds. He could of course have submitted his comments to the
journal, which is the property of the British Sociological Association and as
such is independently edited by a rotating editorial teams. This ensures a
degree of openness lacking in some other journals which belong rather evidently
to particular schools of thought, institutions, discourses – take your pick – and
tend to endorse limited intellectual positions. Mills seems to think that
‘think pieces’ don’t belong in journals. I and the editors disagree, feeling I
assume that arguments about the nature of disciplines and fields are part of a
programme of scholarship which not only informs the conduct and interpretation
of the results of empirical research, but is an essential foundation for claims
as to the nature of reality made on the basis of such research. Argument about
these things matters and it is better for the argument to be as open and public
as possible. I can’t speak for the editors of Sociology but my guess is that they would have welcomed an open
response.
Form aside, let us turn to substance. Mills
took exception to my arguments on the following grounds[ii]:
·
My agreement with Abbott in
rejecting the view that disembodied variables have causal powers.
· My assertion that the
differences which matter in the social world are differences of kind rather
than differences of degree. The consequent of this is that the forms of
measurement which matter are about kinds, sometimes about ordered kinds, but
not about incremental differences and that hence we should focus on the nominal
and the ordinal rather than the continuous.
· My dismissal of the capacity of
conventional linear models for dealing with non-linear i.e. non-incremental
forms of change.
He also took exception to my dismissal,
conventional in the extreme these days I have to say, of the character of
econometric modelling based on linear forms and the assertion of equilibrium.
If you want to pursue that argument I refer you to the work of Paul Ormerod and
Geoffrey Hodgson. Conventional
neo-classical economics is not just intellectual rubbish. It has served a
singularly malignant purpose underpinning the assertion of market rationality
against all other understanding, a significant factor in the creation of the
mess we are in at the beginning of the 21st Century.
Let us begin with ‘variables’. Abbott was
not indulging in rhetoric. He had looked at the character of causal explanation
presented in quantitative articles in the major US sociology journals and saw
that causal powers were assigned to variables extracted both from the real
‘cases’ (cases are both real and constructed – see Charles Ragin’s seminal
discussion of the process of casing) and in general from any sort of context.
Ragin’s careful discussion of set-theoretic explanation, the general form of
actual causal explanations in sociology, is explicitly founded around a
rejection of the crude social physics which looks at variables as the having
causal power, with in terms of his understanding, the outcome of cause being the state of
relevant systems at whatever level. His development of Qualitative Comparative
Analysis was intended exactly as the basis for a quantitative causal approach
which did not begin with reified and abstracted variables. To say this is not
to dismiss measurement of attributes. To take Mills’ example, we might very
well use the attribute of ‘authoritarian’ however we operationalize it as an
element in describing a case but we will never see how that works out in terms
of causal outcomes without looking at how the cases / individuals who are
authoritarian function in contexts and as whole persons with other important
attributes being of significance in determining (a word here used to mean
setting the limits of possibility rather than exact specification) outcomes.
This point can be made more clearly if we
take attributes of institutions, for example secondary schools, and relate
these attribute sets to outcomes, for example the aggregate achievement of
cohorts of children passing through these schools.[iii]
Here we find complex causation in operation in spades. Mills’ considers that
complex causation can be handled in linear models by the specification of
interaction terms. Indeed it can – the specification of interaction terms is
the tribute paid by linear modelling to complex causation – a point made long
ago by Cathie Marsh in The Survey Method.
However, this is not often done – examples if you disagree Mr Mills - and
in any event handling complex causality by specifying interaction terms is
somewhat akin to making a duck by teaching a chicken how to swim. It is clumsy,
leads to difficulties of interpretation, and at very best produces a
model which most parsimoniously reproduces an approximation of the data pattern
displayed in the data set. You can
produce a sort of model of complex causation but the singular is
extremely important.
What is lost in that kind of modelling is
the actual variety of causal pathways towards outcomes. The (parsimonious) model that fits the
data cannot handle equifinality, a term due to Ludwig Von Bertalanffy and
indicating that in an open system there are many pathways towards a given state
of the system – in plain (if odd)
English, there is more than one way to skin a cat. An example can illustrate. I recently explored the multiple and
different ways in which individuals recorded in the British Household Survey
moved among income deciles over time (Byrne:
Getting up - Staying
Up? - Exploring Trajectories in Household Incomes Between 1992 and 2006 Sociological Research Online, 17 (2) 8 2012). I used the truth
table which can be generated from QCA softward. Jenkins who conducted a similar
exploration (JENKINS,
S.P. (2011) Changing Fortunes Oxford: Oxford University Press) but used
statistical modelling methods noted the limitations of his approach: ' … most descriptions
focus on some average experience … They do not reveal the diversity of income
trajectories, even among individuals with similar characteristics.' (2011: 15). Exactly – when we look at any kind of
mobility what confronts us is as Jenkins put it, a spaghetti tangle of
trajectories – lots of them. I found 388 configurations associating income
decile with an attribute set. Few of these were wholly resolved – that is the configuration was not associated with only one
decile band - but the ordinal pattern
was very strong. This is equifinality and it is what confronts us in reality.
Reducing it to a model in which discrete variables have causal powers is wrong.
Let us
turn to ‘differences that matter’. I contend that these are differences of
kind, even if of ordered kind, rather than of degree. What matters is, to
deploy a useful expression, the transformation of quantity into quality. At the
level of social systems this is what concerns sociology - how capitalism emerged from its precursors –
but this does not just apply at the level of the macro. I am very interested in
the transformation – note that word: to transform is to change the kind of
thing a system is – of industrial city regions as they have gone through
processes of profound deindustrialization. We can and should map such
transformations quantitatively. We can also see that the postindustrial
condition of city regions is actually variable (to use that word as the
adjective it should be). Malmo is not the same as Liverpool. Cleveland Ohio
differs from Cleveland (Teesside) UK. Methods of quantitative taxonomy deployed
over time are very useful here. The same holds for individuals. Wendy Dyer’s
careful mapping of the trajectories of individuals through a custody diversion
process in criminal justice is precisely concerned with different kinds
of outcome over time.
Ragin and
Rihoux put this well:
‘ … policy researchers,
especially those concerned with social as opposed to economic policy, are often
more interested in different kinds of cases and their different fates than they
are in the extent of the net causal
effect of a variable across a large encompassing population of observations.
After all, a common goal of social policy is to make decisive interventions,
not to move average levels or rates up or down by some miniscule fraction.’ (Ragin,
C. & B. Rihoux (2004) 'Qualitative Comparative Analysis (QCA): State of the
Art and Prospects' Qualitative Methods. Newsletter of the American
Political Science Association Organized Section on Qualitative Methods, 2
2 3-13 18)
Ragin’s important discussion of calibration
comes into play here. If the ideal form of measurement is always considered to
be continuous / ratio scale / scalar (the baleful legacy of Pearson’s misguided
notion of tetrachoric variation) then we make serious errors. Shakespeare’s
seven ages of man makes the same point. Just because we can count a thing in a
continuous way does not mean that the valid measurement for our purposes is
continuous. Frankly I was surprised that Mills, whose empirical work is largely
in relation to social mobility, took exception to this specification. After all
the whole point of social mobility studies is to explore trajectories which
result in either similarity or difference, ordered categories but categories
nonetheless.
Finally let me deal with the ability of
conventional modelling methods to handle non-linearity. In the original article
I noted that they do attempt to do this but if handling complex causation
through specification of interaction terms can be likened to making a duck by
teaching a chicken how to swim then the appropriate comparison here is making a
swan by attempting the same training for a turkey. What these methods can do is
generate threshold descriptions where linear relation change form. Techniques
such as multivariate adaptive regression splines may through the use of hinge
functions be able to describe non linear transformations in the value of
individual variables but are not of any real use in relation to transformations
in complex systems. This is why
econo-physicists generally use different approaches - despite the limitations
of what is still restricted complexity they do a lot better than econometricians
at modelling economic reality. OK – in terms of specification of the form of a
function, approaches such as splines address non-linearity but what is
important for us is the system, not the function. They deal very poorly with
system change, not least because they work best with outcomes measured at a
continuous level. They can map changes in a dependent continuous variable but do
not adequately handle changes in overall system state.
Frankly the Gulbenkian Commission on the
Future of the Social Sciences got it bang to rights when they noted the
relative failure of the quantitative projects of Sociology, Economics and
Psychology. The reductionist logic which has underpinned the quantitative
programmes in these disciplines has never achieved causal adequacy precisely
because it cannot cope with emergence. Quantitative description has been very
valuable but what is required is radical recasting of our causal approaches. In
the case of Economics whilst the absurd pretensions of the neo-classical programme
in Economics are not the only origin of the mess we are in, they have certainly
been part of the causal complex. In Sociology the damage has been much less. A
lot of paper has been wasted on the presentation of the banal and the trivial
but we are going web-based and electrons are cheaper than trees.
Note that as I made very clear in the
original article to say this is not to dismiss quantitative work. It is instead
a challenge to conventional reductionist academic work. My article was a
response to the benchmarking report on British Sociology. Although much of that
report was sensible and uncontroversial, the comments made about the
quantitative were anything but. They have to set in a context where laudable
efforts to address the mathematical inadequacies of many UK social science
undergraduates, graduate students and even academics, have become inter-twined
with a banal and frankly stupid project of ‘getting Sociologists, Social Policy
people, Anthropologists etc. to imitate the quantitative successes of Economics
and Psychology’. What successes? OK Psychology has more sophistication and use
than neo-classical Economics but one of its major approaches, the Randomized
Controlled Trial, is now being pushed as the panacea in the evaluation of
complex social interventions. The Chief Executive of the ESRC recently asserted
at a meeting I attended that what he was pushing for was more RCTs in social
policy. Frankly I think that this demonstrates a methodological sophistication
which could be written out with a flagpole in letters four foot high on a
flea’s arse but there you go. The Nuffield foundation is pushing the same
agenda just as there is developing in medicine and informed critique of the RCT
and a decent understanding of the limitations of the approach. There is a game afoot and it needs kyboshing
before it gets out of hand.
One concluding point – the Benchmarking
exercise looked at UK Sociology but it did not look at the application of
sociological methods and understanding across the whole range of applied
fields. This reflects the salience of disciplinary silos in the contemporary
academy. I don’t care if people do interesting social simulations in business
schools or epidemiology and publish these anywhere appropriate for that work. I
have plenty to say on the limitations of simulations as presently constructed
but they can, sort of, handle emergence and have great potential. The
Gulbenkian Report called for an opening of the social sciences. That is the way
to go and a quantitative programme which is isomorphic with complex social
reality will help to take us in that direction.
[i] ‘UK Sociology and Quantitative Methods: Are We as Weak as They
Think? Or Are They Barking up the Wrong Tree?’ Sociology 46(1) 13–24 2012
[ii] I am ignoring some trivial
misinterpretations of what I was saying.
[iii] By the way we often have data for all the
cases we are interested in as opposed to samples of them. This is particularly
the case for institutional sets e.g. English secondary schools. Inference from
samples is important – making statements about the whole set on the basis of
information for part of the set, but in a world of big data and electronic
records we are often, perhaps more often than not, working with the whole
rather than the part. I should also note
that although for example % of children obtaining 5 A-C GCSEs can be measured
as continuous variable it is far more useful to classify schools in terms of a
range of attainment measures and use the resultant ordered categories to
explore causation in relation to category membership.
Posts
