David Byrne on Mills on Byrne on quantitative methods

I believe in genuinely open discussion, so when David Byrne contacted me about my post on  his Sociology article on quantitative methods I offered him the opportunity to make a reply in a guest post. So here it is, a new departure in Oxford Sociology, the first guest post. I'll be responding in due course (you wouldn't, gentle reader,  expect anything less).

  

David Byrne

Reply to Colin Mills

Colin Mills took exception to my article in Sociology[i] on a variety of grounds. He could of course have submitted his comments to the journal, which is the property of the British Sociological Association and as such is independently edited by a rotating editorial teams. This ensures a degree of openness lacking in some other journals which belong rather evidently to particular schools of thought, institutions, discourses – take your pick – and tend to endorse limited intellectual positions. Mills seems to think that ‘think pieces’ don’t belong in journals. I and the editors disagree, feeling I assume that arguments about the nature of disciplines and fields are part of a programme of scholarship which not only informs the conduct and interpretation of the results of empirical research, but is an essential foundation for claims as to the nature of reality made on the basis of such research. Argument about these things matters and it is better for the argument to be as open and public as possible. I can’t speak for the editors of Sociology but my guess is that they would have welcomed an open response.

Form aside, let us turn to substance. Mills took exception to my arguments on the following grounds[ii]:

·              My agreement with Abbott in rejecting the view that disembodied variables have causal powers.

·      My assertion that the differences which matter in the social world are differences of kind rather than differences of degree. The consequent of this is that the forms of measurement which matter are about kinds, sometimes about ordered kinds, but not about incremental differences and that hence we should focus on the nominal and the ordinal rather than the continuous. 

·       My dismissal of the capacity of conventional linear models for dealing with non-linear i.e. non-incremental forms of change.

He also took exception to my dismissal, conventional in the extreme these days I have to say, of the character of econometric modelling based on linear forms and the assertion of equilibrium. If you want to pursue that argument I refer you to the work of Paul Ormerod and Geoffrey Hodgson.  Conventional neo-classical economics is not just intellectual rubbish. It has served a singularly malignant purpose underpinning the assertion of market rationality against all other understanding, a significant factor in the creation of the mess we are in at the beginning of the 21^st Century.

Let us begin with ‘variables’. Abbott was not indulging in rhetoric. He had looked at the character of causal explanation presented in quantitative articles in the major US sociology journals and saw that causal powers were assigned to variables extracted both from the real ‘cases’ (cases are both real and constructed – see Charles Ragin’s seminal discussion of the process of casing) and in general from any sort of context. Ragin’s careful discussion of set-theoretic explanation, the general form of actual causal explanations in sociology, is explicitly founded around a rejection of the crude social physics which looks at variables as the having causal power, with in terms of his understanding,  the outcome of cause being the state of relevant systems at whatever level. His development of Qualitative Comparative Analysis was intended exactly as the basis for a quantitative causal approach which did not begin with reified and abstracted variables. To say this is not to dismiss measurement of attributes. To take Mills’ example, we might very well use the attribute of ‘authoritarian’ however we operationalize it as an element in describing a case but we will never see how that works out in terms of causal outcomes without looking at how the cases / individuals who are authoritarian function in contexts and as whole persons with other important attributes being of significance in determining (a word here used to mean setting the limits of possibility rather than exact specification) outcomes. 

This point can be made more clearly if we take attributes of institutions, for example secondary schools, and relate these attribute sets to outcomes, for example the aggregate achievement of cohorts of children passing through these schools.[iii] Here we find complex causation in operation in spades. Mills’ considers that complex causation can be handled in linear models by the specification of interaction terms. Indeed it can – the specification of interaction terms is the tribute paid by linear modelling to complex causation – a point made long ago by Cathie Marsh in The Survey Method. However, this is not often done – examples if you disagree Mr Mills - and in any event handling complex causality by specifying interaction terms is somewhat akin to making a duck by teaching a chicken how to swim. It is clumsy, leads to difficulties of interpretation, and at very best produces a model which most parsimoniously reproduces an approximation of the data pattern displayed in the data set.  You can produce a sort of model of complex causation but the singular is extremely important.

What is lost in that kind of modelling is the actual variety of causal pathways towards outcomes.  The (parsimonious) model that fits the data cannot handle equifinality, a term due to Ludwig Von Bertalanffy and indicating that in an open system there are many pathways towards a given state of the system – in  plain (if odd) English, there is more than one way to skin a cat. An example can illustrate.  I recently explored the multiple and different ways in which individuals recorded in the British Household Survey moved among income deciles over time (Byrne:  Getting up - Staying Up? - Exploring Trajectories in Household Incomes Between 1992 and 2006 Sociological Research Online, 17 (2) 8 2012). I used the truth table which can be generated from QCA softward. Jenkins who conducted a similar exploration (JENKINS, S.P. (2011) Changing Fortunes Oxford: Oxford University Press) but used statistical modelling methods noted the limitations of his approach: ' … most descriptions focus on some average experience … They do not reveal the diversity of income trajectories, even among individuals with similar characteristics.' (2011: 15). Exactly – when we look at any kind of mobility what confronts us is as Jenkins put it, a spaghetti tangle of trajectories – lots of them. I found 388 configurations associating income decile with an attribute set. Few of these were wholly resolved – that is the  configuration was not associated with only one decile band  - but the ordinal pattern was very strong. This is equifinality and it is what confronts us in reality. Reducing it to a model in which discrete variables have causal powers is wrong.

Let us turn to ‘differences that matter’. I contend that these are differences of kind, even if of ordered kind, rather than of degree. What matters is, to deploy a useful expression, the transformation of quantity into quality. At the level of social systems this is what concerns sociology -  how capitalism emerged from its precursors – but this does not just apply at the level of the macro. I am very interested in the transformation – note that word: to transform is to change the kind of thing a system is – of industrial city regions as they have gone through processes of profound deindustrialization. We can and should map such transformations quantitatively. We can also see that the postindustrial condition of city regions is actually variable (to use that word as the adjective it should be). Malmo is not the same as Liverpool. Cleveland Ohio differs from Cleveland (Teesside) UK. Methods of quantitative taxonomy deployed over time are very useful here. The same holds for individuals. Wendy Dyer’s careful mapping of the trajectories of individuals through a custody diversion process in criminal justice is precisely concerned with different kinds of outcome over time.

Ragin and Rihoux put this well:

‘ …  policy researchers, especially those concerned with social as opposed to economic policy, are often more interested in different kinds of cases and their different fates than they are  in the extent of the net causal effect of a variable across a large encompassing population of observations. After all, a common goal of social policy is to make decisive interventions, not to move average levels or rates up or down by some miniscule fraction.’ (Ragin, C. & B. Rihoux (2004) 'Qualitative Comparative Analysis (QCA): State of the Art and Prospects' Qualitative Methods. Newsletter of the American Political Science Association Organized Section on Qualitative Methods, 2 2 3-13 18)

Ragin’s important discussion of calibration comes into play here. If the ideal form of measurement is always considered to be continuous / ratio scale / scalar (the baleful legacy of Pearson’s misguided notion of tetrachoric variation) then we make serious errors. Shakespeare’s seven ages of man makes the same point. Just because we can count a thing in a continuous way does not mean that the valid measurement for our purposes is continuous. Frankly I was surprised that Mills, whose empirical work is largely in relation to social mobility, took exception to this specification. After all the whole point of social mobility studies is to explore trajectories which result in either similarity or difference, ordered categories but categories nonetheless.

Finally let me deal with the ability of conventional modelling methods to handle non-linearity. In the original article I noted that they do attempt to do this but if handling complex causation through specification of interaction terms can be likened to making a duck by teaching a chicken how to swim then the appropriate comparison here is making a swan by attempting the same training for a turkey. What these methods can do is generate threshold descriptions where linear relation change form. Techniques such as multivariate adaptive regression splines may through the use of hinge functions be able to describe non linear transformations in the value of individual variables but are not of any real use in relation to transformations in complex  systems. This is why econo-physicists generally use different approaches - despite the limitations of what is still restricted complexity they do a lot better than econometricians at modelling economic reality. OK – in terms of specification of the form of a function,  approaches such as   splines address non-linearity but what is important for us is the system, not the function. They deal very poorly with system change, not least because they work best with outcomes measured at a continuous level. They can map changes in a dependent continuous variable but do not adequately handle changes in overall system state. 

Frankly the Gulbenkian Commission on the Future of the Social Sciences got it bang to rights when they noted the relative failure of the quantitative projects of Sociology, Economics and Psychology. The reductionist logic which has underpinned the quantitative programmes in these disciplines has never achieved causal adequacy precisely because it cannot cope with emergence. Quantitative description has been very valuable but what is required is radical recasting of our causal approaches. In the case of Economics whilst the absurd pretensions of the neo-classical programme in Economics are not the only origin of the mess we are in, they have certainly been part of the causal complex. In Sociology the damage has been much less. A lot of paper has been wasted on the presentation of the banal and the trivial but we are going web-based and electrons are cheaper than trees.  

Note that as I made very clear in the original article to say this is not to dismiss quantitative work. It is instead a challenge to conventional reductionist academic work. My article was a response to the benchmarking report on British Sociology. Although much of that report was sensible and uncontroversial, the comments made about the quantitative were anything but. They have to set in a context where laudable efforts to address the mathematical inadequacies of many UK social science undergraduates, graduate students and even academics, have become inter-twined with a banal and frankly stupid project of ‘getting Sociologists, Social Policy people, Anthropologists etc. to imitate the quantitative successes of Economics and Psychology’. What successes? OK Psychology has more sophistication and use than neo-classical Economics but one of its major approaches, the Randomized Controlled Trial, is now being pushed as the panacea in the evaluation of complex social interventions. The Chief Executive of the ESRC recently asserted at a meeting I attended that what he was pushing for was more RCTs in social policy. Frankly I think that this demonstrates a methodological sophistication which could be written out with a flagpole in letters four foot high on a flea’s arse but there you go. The Nuffield foundation is pushing the same agenda just as there is developing in medicine and informed critique of the RCT and a decent understanding of the limitations of the approach.  There is a game afoot and it needs kyboshing before it gets out of hand.

One concluding point – the Benchmarking exercise looked at UK Sociology but it did not look at the application of sociological methods and understanding across the whole range of applied fields. This reflects the salience of disciplinary silos in the contemporary academy. I don’t care if people do interesting social simulations in business schools or epidemiology and publish these anywhere appropriate for that work. I have plenty to say on the limitations of simulations as presently constructed but they can, sort of, handle emergence and have great potential. The Gulbenkian Report called for an opening of the social sciences. That is the way to go and a quantitative programme which is isomorphic with complex social reality will help to take us in that direction.

---

[i] ‘UK Sociology and Quantitative Methods: Are We as Weak as They Think? Or Are They Barking up the Wrong Tree?’ Sociology 46(1) 13–24 2012

[ii] I am ignoring some trivial misinterpretations of what I was saying.

[iii] By the way we often have data for all the cases we are interested in as opposed to samples of them. This is particularly the case for institutional sets e.g. English secondary schools. Inference from samples is important – making statements about the whole set on the basis of information for part of the set, but in a world of big data and electronic records we are often, perhaps more often than not, working with the whole rather than the part.  I should also note that although for example % of children obtaining 5 A-C GCSEs can be measured as continuous variable it is far more useful to classify schools in terms of a range of attainment measures and use the resultant ordered categories to explore causation in relation to category membership.