In the preface to Designing Social Inquiry, King, Keohane and Verba describe the spirit of their collaboration in the following way:
"... our rules of engagement meant that "agreeing to disagree" and compromising were high crimes. If one of us was not truly convinced of a point, we took it as our obligation to continue the debate."
I've always thought this a fine ideal to aspire to. I want to know exactly why it is that others disagree with me for only then can I hope to persuade them that I am right or, when I am wrong, learn how I ever came to believe such foolish things.
If the disagreement is about matters of fact, then, once we have agreed on what needs to be established, all we have to do is look and we'll find out who spoke most correctly. Not all disagreements however are about facts. A lot of the time they are about what, for want of a better word, I'll call meta-theory - for instance the claim that X is "more important" (in some sense) than Y or that X is "more interesting" than Y.
It turns out that facts are not wholly irrelevant to making judgements about these sorts of things, though they can't be decisive. But it is also not the case that we can't have a certain amount of reasoned argument about them, we can and do. The issue is this: at some point we reach bedrock and if David prefers pushpin to Pushkin and all that follows from that while Colin prefers the reverse then we've gone as far as we (and probably our readers) can usefully go.
It turns out that facts are not wholly irrelevant to making judgements about these sorts of things, though they can't be decisive. But it is also not the case that we can't have a certain amount of reasoned argument about them, we can and do. The issue is this: at some point we reach bedrock and if David prefers pushpin to Pushkin and all that follows from that while Colin prefers the reverse then we've gone as far as we (and probably our readers) can usefully go.
In what follows I'll try to distinguish as clearly as I can the issues that divide us in terms of matters of fact, belief and value. Faute de mieux I'll take up Byrne's points of contention in roughly the order he raises them.
1. Byrne believes that scholarly journals are an appropriate place for 'think pieces'. First a matter of fact: I didn't use the term 'think piece', what I said was: "I doubt that refereed journals are an appropriate place for opinion pieces...". Opinions are, well, opinions and may embody more or less thinking. Who gives a damn about the opinions of a bucket load of referees about opinions? What possible scholarly tests could they be applying? Whatever these were they couldn't, in this case, be, in my opinion, rigorous or coherent, but then again, that's just my opinion. People sweat blood to carry out genuine research and it must be galling for them to see the publication of the hard won fruit of their effort delayed by the ceding of a precious journal slot to opinion.
But this is not a particularly important point of disagreement though Byrne continues to express quite illogical views about it. Try to square these: "... it is better for the argument to be as open and public
as possible" whilst at the same time advocating that it is better to have this conversation in the pages of an academic journal that is locked behind a pay-wall. How "open and public" is that? or is Byrne using "open and public" in some special sense that the rest of us aren't privy to? I don't know and it doesn't matter, for there are much more important things to disagree about than the forum of the discussion.
2. If we are going to have a rational discussion we have to agree to stick to the point. That means not introducing irrelevancies or attributing to each other views that haven't been expressed. What my or Byrne's views are about neo-classical economics (or any other sort of economics) or the role of equilibrium in economic analysis are completely irrelevant. Neo-classical economics is, very obviously, not econometrics and neither I, nor Byrne as far as I can see, had anything whatsoever to say about notions of equilibrium, so how I could take exception to his views on it beats me. All I can conclude is that Byrne seems to have a very active fantasy life.
For what it's worth, my preference is to let the economists and the econometricians take care of themselves. I'm lucky enough to work in an institution where I regularly rub shoulders with colleagues in these fields and I know for a fact who would come off the worse if I were foolish enough to attempt to tell the likes of David Hendry, Neil Shepherd, Paul Klemperer or Kevin Roberts (just to pick a few names at random) how to do their business. I don't tell them how to do econometrics/economics and they don't tell me how to do sociology.
My point was and is very simple. If Byrne wants to tell us that econometrics is fatally flawed, and now that neo-classical economics is "intellectual rubbish", I'd be more impressed if he stepped down from the pulpit and stopped preaching to the choir (it would also help if he had a few serious arguments). When he's gone eyeball to eyeball with Hendry, Shepherd, Granger, Pagan, Robinson, Heckman and Blundell and routed the whole assembly, then I'll begin to take him seriously and listen, if there is anything left to listen to.
My point was and is very simple. If Byrne wants to tell us that econometrics is fatally flawed, and now that neo-classical economics is "intellectual rubbish", I'd be more impressed if he stepped down from the pulpit and stopped preaching to the choir (it would also help if he had a few serious arguments). When he's gone eyeball to eyeball with Hendry, Shepherd, Granger, Pagan, Robinson, Heckman and Blundell and routed the whole assembly, then I'll begin to take him seriously and listen, if there is anything left to listen to.
3. Byrne takes seriously Abbott on so called "variable sociology" and I don't. Anyone who seriously believes that in the human realm variables rove around the world and do things is either a lunatic or an idiot. I don't believe that and neither does anyone else in my neck of the woods. I think I put this pretty clearly in my earlier post and don't see any reason to repeat myself.
Byrne is also evidently impressed by Ragin's thoughts on cases and I'm unimpressed. I don't deny that sometimes defining what is a case and what it is a case of is not straightforward - think about revolutions for example - (and sometimes it is very straightforward), but personally I don't find Ragin's appercus rewarding or of much practical help. Neither do I find the rhetoric of "whole cases" in their contexts especially appealing. What, everything, with nothing left out? Regardless of relevance? Or is some selectivity to be smuggled in at the back door? Surely it must be otherwise it is a recipe for turning "whole case" sociologists into a species of Casaubons, always packing their bags but never going on a journey. Though given the lack of startling empirical insights and revelations from the QCA fad maybe I'm closer to the mark than I thought.
Social science, if you are actually going to do it rather than just talk about it, requires judgments about what to include and what to leave out just as it requires judgements about the right amount of context to retain. Sometimes context is important, sometimes it is irrelevant. One can only make sensible judgments about these things by examining specifics and since Byrne never gets down to specifics it is difficult to say anything more constructive.
4) On interaction effects. OK, so now we have to get down to some matters of fact (I'll try to avoid outlandish avian similes). Byrne wants citations of interaction effects. I'll give him 50 metres start and tie my legs together by promising not to cite my own work or the work of anyone personally known to me. I'll also not include papers that use logit models or multinomial logit models even though they are special cases of the log-linear model.
First stop, the search function of the online version of the European Sociological Review. I restrict my search to articles published in the last 12 years that report the use of log-linear models and visually inspect the tables for specifications of at least one three-way or higher interaction effect (I further handicap myself by not permitting myself to include implicit interactions such as when separate models are fitted for males and females, different time periods, different countries etc). A search taking about 15 minutes came up with 8 papers (if I'd been easier on myself the number would have been roughly 3 times as large) and that is just in one journal. Given that the fashion for log-linear models was over about 20 years ago and therefore the base-line frequency for articles that use any sort of log-linear models is quite low, I find this impressive (but then I would say that wouldn't I). Most readers probably don't care about the details, but for those that do, I list the papers at the bottom of this post.
So Professor Byrne is that enough, or do you want to raise the bar higher and introduce some further ad hoc qualifications? Or are you willing now to admit that you are just wrong about the facts of the matter? Or do you, in a Frankfurtian spirit, just not care about the facts?
5) Now we get to the bit that gives me no pleasure at all - the dissection of Byrne's piece de resistance: Getting up - Staying Up? - Exploring Trajectories in Household Incomes Between 1992 and 2006 Sociological Research Online, 17 (2) 8 2012). As my old headmaster used to say, this is going to hurt me far more than it is going to hurt you...
5) Now we get to the bit that gives me no pleasure at all - the dissection of Byrne's piece de resistance: Getting up - Staying Up? - Exploring Trajectories in Household Incomes Between 1992 and 2006 Sociological Research Online, 17 (2) 8 2012). As my old headmaster used to say, this is going to hurt me far more than it is going to hurt you...
Frankly I don't believe this article contributes anything to the debate, or indeed to the understanding of income mobility in the UK (but that is another matter). What it does is create a vast smoke screen of irrelevance and trickery. I'll quote Byrne and then put my comments underneath:
"I recently explored the multiple and different ways in which individuals recorded in the British Household Survey moved among income deciles over time."
This is completely misleading. What you actually did was looked at data from the British Household Panel Survey on the movement between 4 arbitrarily chosen partitions of the the household income distribution in 1992 (only one of which was (you claim) a decile, though the label in your excel table seems to suggest that it was actually the top 5%) and 2006 ie two time points.
"I used the truth table which can be generated from QCA software."
That sounds impressive, but actually all you have done (in Table 3) is made an indicator matrix from the dichotomised explanatory variables and then counted the number of rows that fall within each of the categories of the income in 2006 variables and, er..., that's it.
"Jenkins who conducted a similar exploration (JENKINS, S.P. (2011) Changing Fortunes Oxford: Oxford University Press) but used statistical modelling methods noted the limitations of his approach: ' … most descriptions focus on some average experience … They do not reveal the diversity of income trajectories, even among individuals with similar characteristics.' (2011: 15)."
No he didn't (make a similar exploration) and as it happens I'm in a position to know because I served alongside Jenkins throughout 2007 on the National Equality Panel where he presented some preliminary versions of his work. What Jenkins does in fact do is examine longitudinal income trajectories, not movement between two points in time and explicitly reveals the diversity of income trajectories among people with similar characteristics (that's what random effects models do Professor Byrne, so where exactly is the problem with conventional methods?).
"I found 388 configurations...."
Big whoop. You claim to have started with 213 =8192 configurations of the attributes, but actually a large number of these are logically impossible (a case can't simultaneously be in the bottom 50% of the 1992 income distribution and in the top 10%, can't simultaneously have no partner and a partner in work, can't not be in the Goldthorpe Service class and have a managerial job). Your calculation, Professor Byrne, is wrong (credit to my colleague Christiaan Monden for drawing this to my attention and shame on SRO referees for not noticing or caring). It would be right if you were working with 13 dichotomies, but you aren't, two of your factors (variables or whatever you want to call them) are polytomous. The appropriate calculation would be 4x3x2x2x2x2x2x2=768. Even this isn't quite right because by definition a manager can only be a member of Goldthorpe's service class. I would have more confidence in your pronouncements about econometrics if you were able to get the basics right.
768 is still a lot of possible combinations of values, but so what? We can arbitrarily increase these by chopping each attribute more and more finely, adding more attributes etc etc, but to what end? We can even treat each individual as a case (is that "singular" enough for you?) after all we have multiple observations on each individual and estimate person specific income slopes (with more or less any functional form you care to specify). And what do we learn? That things are very complicated? 213 complicated? Come on, stop trying to pull the wool over our eyes. We knew that already. What we want to extract from data is structure, not idiosyncrasy or measurement error.
I could go on. I won't. It's embarrassing.
6) "Let us turn to ‘differences that matter’. I contend that these are differences of kind, even if of ordered kind, rather than of degree." So say you Professor Byrne but what, apart from assertion, lies behind that? Matters for what? Again we can't have a sensible discussion about this without getting down to specifics. As I've said to you before in an offline conversation:
"If I wanted to make a prediction model for weather systems I wouldn't use the mathematics of the GLM and if I did it wouldn't work. But that says nothing significant about the usefulness of the GLM. To judge the usefulness of a tool you have to specify what the nature of the task is and if you don't then it is just cheap talk. If I want to cut a piece of wood I use a saw not a hammer.
If all you meant to say was that you can't do the equivalent of weather science with the GLM then who are you arguing against? I doubt though that the average Sociology reader will take your assertions in that way. "
If you want to get serious and start talking about systems of differential equations with feedback loops, tipping points and all the rest of it then let's get down to brass tacks. I've got good friends whose intellects I respect who do system dynamics. I've nothing against thinking in terms of systems and I've seen some interesting sociological applications (for example this site on church growth modelling) but we need to do this at the level of specifics in the context of particular applications, not at the level of rhetoric and windy generalization. It would also help if you managed to get clear some basic distinctions - like the difference between linearity and continuity but perhaps we'll save that for another time.
7) It would also help if you managed to read what I actually wrote correctly and desisted from attributing to me things that I clearly have not done or said. Take for instance this gem:
"Just because we can count a thing in a continuous way does not mean that the valid measurement for our purposes is continuous. Frankly I was surprised that Mills, whose empirical work is largely in relation to social mobility, took exception to this specification. After all the whole point of social mobility studies is to explore trajectories which result in either similarity or difference, ordered categories but categories nonetheless."
Counting is by the way discrete (sorry to be pedantic) - the give away is the convention that we count things using integers- and we measure things continuously using numbers on the real line, but let's not get bogged down in terminology. The more important point is: where, exactly, have I taken "exception to this specification"? I don't even know what "this specification" is let alone taken exception to it. Maybe this sort of rhetorical BS goes down well with the BSA home crowd, but it doesn't cut the mustard in my patch and I doubt if it will fool many of my readers. There are lots of people who can give me lessons about how to do research on social mobility, but somehow, on this showing, I doubt that David Byrne is one of them.
8) And finally, RCTs. This has nothing to do with my original post, but since Byrne brought it up...The blanket dismissal of RCTs is ridiculous. There are plenty of serious and thoughtful people who write about the limitations of RCTs - Cartwright, Worrall and from a rather different perspective Heckman, but none of these are remotely as dismissive as Byrne. If your intellectual project is to estimate the effect of a cause, then an RCT is one of the major ways of achieving that aim (another is to use observational data and an appropriate instrument). It, of course, assumes that you have already identified the causal variable of interest, can randomize (or simulate randomization) and/or condition on any exogenous confounders. For some problems this is exactly what you want regardless of whether you are interested in estimating the average causal effect or (where the action is these days) examining the heterogeneity of causal effects. Everyone who is a serious user of RCTs knows about SUTVA and what that implies for the generalizability of a causal estimate beyond the limits of changes at the margin.
Not all problems in the social sciences and certainly not all sociological problems can be addressed by RCTs and nobody has ever said they can be. If your intellectual problem is to figure out the causes of an effect then an RCT isn't going to help you. There are smart people thinking about these things, for example: Dyson, Tim and Bhrolcháin, M (2007) On causation in demography Population and development review, 33 (1). 1-36 and there is a long epidemiological tradition (think smoking and lung cancer) to draw on. What seems to be becoming clear is that Mackie's INUS conditions aren't all that helpful in practice. I'd be delighted to discuss all of this with Professor Byrne if he can manage to come down to this level of specificity and stick to the point.
Where does this leave us? Well I'll let the readers judge. To my eye Byrne comes over all tame at the end of his response and wants us to see him as a friend of quantification in sociology. If we are on the same side I'm left with a feeling akin to that of the Duke of Wellington when inspecting the riff-raff that comprised his army before Waterloo: "I don't know what they do to the enemy, but by God they frighten me!".
"I found 388 configurations...."
Big whoop. You claim to have started with 213 =8192 configurations of the attributes, but actually a large number of these are logically impossible (a case can't simultaneously be in the bottom 50% of the 1992 income distribution and in the top 10%, can't simultaneously have no partner and a partner in work, can't not be in the Goldthorpe Service class and have a managerial job). Your calculation, Professor Byrne, is wrong (credit to my colleague Christiaan Monden for drawing this to my attention and shame on SRO referees for not noticing or caring). It would be right if you were working with 13 dichotomies, but you aren't, two of your factors (variables or whatever you want to call them) are polytomous. The appropriate calculation would be 4x3x2x2x2x2x2x2=768. Even this isn't quite right because by definition a manager can only be a member of Goldthorpe's service class. I would have more confidence in your pronouncements about econometrics if you were able to get the basics right.
768 is still a lot of possible combinations of values, but so what? We can arbitrarily increase these by chopping each attribute more and more finely, adding more attributes etc etc, but to what end? We can even treat each individual as a case (is that "singular" enough for you?) after all we have multiple observations on each individual and estimate person specific income slopes (with more or less any functional form you care to specify). And what do we learn? That things are very complicated? 213 complicated? Come on, stop trying to pull the wool over our eyes. We knew that already. What we want to extract from data is structure, not idiosyncrasy or measurement error.
I could go on. I won't. It's embarrassing.
6) "Let us turn to ‘differences that matter’. I contend that these are differences of kind, even if of ordered kind, rather than of degree." So say you Professor Byrne but what, apart from assertion, lies behind that? Matters for what? Again we can't have a sensible discussion about this without getting down to specifics. As I've said to you before in an offline conversation:
"If I wanted to make a prediction model for weather systems I wouldn't use the mathematics of the GLM and if I did it wouldn't work. But that says nothing significant about the usefulness of the GLM. To judge the usefulness of a tool you have to specify what the nature of the task is and if you don't then it is just cheap talk. If I want to cut a piece of wood I use a saw not a hammer.
If all you meant to say was that you can't do the equivalent of weather science with the GLM then who are you arguing against? I doubt though that the average Sociology reader will take your assertions in that way. "
If you want to get serious and start talking about systems of differential equations with feedback loops, tipping points and all the rest of it then let's get down to brass tacks. I've got good friends whose intellects I respect who do system dynamics. I've nothing against thinking in terms of systems and I've seen some interesting sociological applications (for example this site on church growth modelling) but we need to do this at the level of specifics in the context of particular applications, not at the level of rhetoric and windy generalization. It would also help if you managed to get clear some basic distinctions - like the difference between linearity and continuity but perhaps we'll save that for another time.
7) It would also help if you managed to read what I actually wrote correctly and desisted from attributing to me things that I clearly have not done or said. Take for instance this gem:
"Just because we can count a thing in a continuous way does not mean that the valid measurement for our purposes is continuous. Frankly I was surprised that Mills, whose empirical work is largely in relation to social mobility, took exception to this specification. After all the whole point of social mobility studies is to explore trajectories which result in either similarity or difference, ordered categories but categories nonetheless."
Counting is by the way discrete (sorry to be pedantic) - the give away is the convention that we count things using integers- and we measure things continuously using numbers on the real line, but let's not get bogged down in terminology. The more important point is: where, exactly, have I taken "exception to this specification"? I don't even know what "this specification" is let alone taken exception to it. Maybe this sort of rhetorical BS goes down well with the BSA home crowd, but it doesn't cut the mustard in my patch and I doubt if it will fool many of my readers. There are lots of people who can give me lessons about how to do research on social mobility, but somehow, on this showing, I doubt that David Byrne is one of them.
8) And finally, RCTs. This has nothing to do with my original post, but since Byrne brought it up...The blanket dismissal of RCTs is ridiculous. There are plenty of serious and thoughtful people who write about the limitations of RCTs - Cartwright, Worrall and from a rather different perspective Heckman, but none of these are remotely as dismissive as Byrne. If your intellectual project is to estimate the effect of a cause, then an RCT is one of the major ways of achieving that aim (another is to use observational data and an appropriate instrument). It, of course, assumes that you have already identified the causal variable of interest, can randomize (or simulate randomization) and/or condition on any exogenous confounders. For some problems this is exactly what you want regardless of whether you are interested in estimating the average causal effect or (where the action is these days) examining the heterogeneity of causal effects. Everyone who is a serious user of RCTs knows about SUTVA and what that implies for the generalizability of a causal estimate beyond the limits of changes at the margin.
Not all problems in the social sciences and certainly not all sociological problems can be addressed by RCTs and nobody has ever said they can be. If your intellectual problem is to figure out the causes of an effect then an RCT isn't going to help you. There are smart people thinking about these things, for example: Dyson, Tim and Bhrolcháin, M (2007) On causation in demography Population and development review, 33 (1). 1-36 and there is a long epidemiological tradition (think smoking and lung cancer) to draw on. What seems to be becoming clear is that Mackie's INUS conditions aren't all that helpful in practice. I'd be delighted to discuss all of this with Professor Byrne if he can manage to come down to this level of specificity and stick to the point.
Where does this leave us? Well I'll let the readers judge. To my eye Byrne comes over all tame at the end of his response and wants us to see him as a friend of quantification in sociology. If we are on the same side I'm left with a feeling akin to that of the Duke of Wellington when inspecting the riff-raff that comprised his army before Waterloo: "I don't know what they do to the enemy, but by God they frighten me!".
Tomáš Katrňák, Martin Kreidl,
and Laura Fónadová, Trends
in Educational Assortative Mating in Central Europe: the Czech Republic,
Slovakia, Poland, and Hungary, 1988–2000 Eur Sociol Rev (2006) 22(3):
309-322
Gabriele Ballarino, Fabrizio Bernardi, Miguel Requena, and Hans Schadee, Persistent
Inequalities? Expansion of Education and Class Inequality in Italy and Spain Eur
Sociol Rev (2009) 25(1): 123-138
Ayşe Gündüz‐Hoşgör
and Jeroen Smits, Intermarriage
between Turks and Kurds in Contemporary Turkey: Inter‐ethnic Relations in an Urbanizing Environment Eur
Sociol Rev (2002) 18(4): 417-432
Mieke C.W. Eeckhaut, Bart Van de Putte, Jan R.M. Gerris, and Ad A. Vermulst Analysing
the Effect of Educational Differences between Partners: A
Methodological/Theoretical Comparison Eur Sociol Rev (2013) 29(1):
60-73
Jens Beckert andMark Lutter
The Inequality of Fair Play: Lottery Gambling
and Social Stratification in Germany Eur Sociol Rev (2009) 25(4): 475-488
Jochem Tolsma, Marcel Coenders, and Marcel Lubbers
Trends in Ethnic Educational Inequalities in the
Netherlands: A Cohort Design Eur Sociol Rev (2007) 23(3): 325-339
Jani Erola and Pasi Moisio Social Mobility over Three Generations in
Finland, 1950–2000 Eur Sociol Rev (2007) 23(2): 169-183
Ellu Saar
Changes in Intergenerational Mobility and
Educational Inequality in Estonia: Comparative Analysis of Cohorts Born between
1930 and 1974 Eur Sociol Rev (2010) 26(3): 367-383
Posts
No comments:
Post a Comment