Oxford's Institutional Bias 2

I said I would post again about this so here it is. As I made clear in my last post I don't know what the RELEVANT facts of the matter are about ethnic group variation in access rates to undergraduate degrees at Oxford and I think it is in the public interest for the appropriate information to be made available. I'd go further than that and say that the University should be monitoring admissions, getting a competent person to analyze the data, and putting the results in the public domain. If something is wrong about the way we are doing things then we should know and we should put it right. If we don't, then we only have ourselves to blame if journalists have to resort to FOI requests and then tell only carefully selected bits of the story. In fact we haven't been entirely negligent - a few years ago the university put quite a bit of money into the Oxford Admissions Study which was conducted by some of my colleagues. It's a great shame that this initiative wasn't carried on.

OK, now to my comments on the Guardian story. As always the Devil is in the details and the details are important  because if we don't pay attention to them we can fail to learn what it is that  data tells us.

1) If we want to learn about the selection process that generated the outcomes, which I assume are accurately described, then you have to make a serious effort to model that process and estimate numbers which correspond to the behaviours that are elicited by the choices, costs, constraints and benefits actually confronting the people involved ie students and admissions tutors. That means, amongst other things, you have to take account only of  information available to the actors at the points at which their choices are made. So, for example, admissions tutors do not choose students on the basis of their actual A level grades. They do have other information that is correlated with grades, but there is also, inevitably, error. So, to make it concrete, I happened to get the highest A level grades that it was possible for somebody taking 3 A levels to get, but at the point of application to universities all I had was a clutch of very mediocre O levels and a good reference from my Head of Sixth Form who took infinite pains in making careful personalized assessments of each of the pupils she wrote for (for which I will be eternally grateful). Despite her best efforts my O level performance would never have got me an Oxbridge offer in any subject. If I had applied and been rejected would that have  demonstrated that Oxbridge admissions tutors were prejudiced against children from the provincial lower middle classes? No of course it wouldn't. That doesn't mean that they weren't (and aren't) but it can't count as evidence in favour of the hypothesis because there is a reasonable alternative: I simply hadn't achieved enough at age 16 and there were a very large number of much better qualified candidates, yet if you control for my A level score it might look as though I'd been hard done by.

2) So controlling for GCSE performance is OK, as is controlling for predicted A level grade, but controlling for actual A level grade is not. Take a look at the Guardian data. Do you notice something odd? Where are the people that got at least 1 B at A level. It surely can't be the case that everyone who applies eventually gets at least 3 As. It would be a remarkable world in which there were no slip ups. Would you imagine that there might be a correlation between GCSE score and not quite getting the grade required? In other words there is an odd kind of sample selection going on here with respect to what really matters ie the information available ex ante.

3) You can't arbitrarily ignore important features of the selection process, ie the fact that colleges play an important role in admissions, that college choice is largely up to the candidates and that candidates are differentially equipped to maximize their chance of entry by making canny choices about which colleges to choose. Now it may be that empirically it turns out that choice of college is irrelevant. If so, all well and good, we've found out something we didn't know, but we can't simply assume that or appeal to official rhetoric - they would say that wouldn't they. If you want to understand the process you have to make some attempt to model the process no matter how difficult that may be. If you think colleges are irrelevant then it is not unreasonable of me to demand that you show me that this is the case. If it is hard then you have to get somebody who is up to the job to extract the maximum possible information from the (imperfect) data to hand.

4) You shouldn't trawl around for "significant" differences. Looking at the Guardian data as a whole what strikes me is that in very many subjects there are no differences worth talking about. In a few subjects there are some differences (but remember my caveats in points 1 & 2 above), medicine is one. Medicine is an important  subject and I would in no way wish to avoid investigating what lies behind these numbers. One of the things though that lies behind significant findings is that if you trawl through enough comparisons you will, just by chance, find some in the direction you are looking for and it is highly likely that the difference you find will be much bigger than the real difference ie unintelligent data-mining runs the risk of exaggerating effect sizes, often quite considerably. Set your p. value at 0.05 and 5% of the time you will find a difference that isn't "really" there. Do this in the context of a big data dredge and the probability is much greater.

So what's the way forward? The obvious answer is that the University should put enough of the right data in the public domain so that a proper analysis can be done on it. Important questions have, quite properly, been raised in the Guardian but we are very far from being in a position to give proper answers to them and the Guardian's own account is far from adequate. That's not to say that when all is said and done there might not be something to it. The fact is that at the moment we just don't know and we shouldn't be leaping to conclusions, particularly when those conclusions seem to imply inappropriate behaviour on the part of some admissions tutors, when all that has to be done is establish properly what the facts of the matter are. Keep the hair shirts on standby, we might yet need them, but first question all the suspects before you name the accused.

Oxford's Institutional Bias

Time for the hair shirts. It's official, (or at least in the Guardian) Oxford appears to discriminate against undergraduate applicants from certain sorts of ethnic backgrounds. It so happens that I have some background information on this story which I'll let you into.

Before I tell my tale, let me make my position clear: 1) I don't know whether there is discrimination; 2) The evidence that has been put in the public domain is interesting, worth discussing, but very far from compelling; 3) I'm not a cheer leader for the Oxford admissions system; 4) I'm a believer in post A level grade admission to all universities, with, wherever feasible, grade threshold criteria (ie a driving test type set up in which everyone who meets the standard is eligible) for admission and a lottery in the event of over subscription. Before you tell me this is utopia, you should know that this has been and for all I know still is the way that the Netherlands handles entry to medical school. So, no interviews and no special tests unless there is an unarguable case for them, say on public interest grounds - ie given the restrictive practices of the BMA we might want to give a sleep deprivation tests to potential medics.

OK, so my story begins on 31st of January when I get the following email out of the blue from somebody called Kurien Parel:

Dr Mr Mills

I am researching for a  Telegraph story regarding what I believe to be a serious ethnic minority bias seen from Oxbridge Admissions statistics. I have found out that ethnic minorities with the same grades are significantly less likely (statistically) to be given an offer to study at Oxford after the interview process. The figures, in my opinion, are quite appalling. In medicine for example a white applicant is around twice as likely to be given an offer as a non-white applicant with the same grade. Indeed in medicine, and in other subjects, white applicants with fewer A*s in A level have significantly higher chance of admission than  non-white applicants.  

I want to know whether you would be interested in talking with me, potentially, to provide your perspective.  I can show you if you are interested some of the actual figures.  

Thanks

Kurien

Odd story for the Telegraph to be running with  is my first thought, so first I check this guy out. Googling the name brings up some hits for a  someone who seems to have done a post-graduate degree at Cambridge and a bit of student journalism for Cambridge's Varsity Online and not much else. Ah well, freelancers have to start somewhere.

I reply in an honest but bland way:

Dear Kurien

I'm afraid I'm not at all involved in undergraduate admissions (Nuffield is a graduate college) so don't have anything  enlightening or knowledge based to say. You would be better off talking to tutors in the u/g colleges who are directly involved in u/g admissions. They are the people that make the decisions that count.

I would be interested to know though what new data you have on Oxbridge admissions. I don't normally comment on anything unless I can examine the primary evidence myself.

all best with your story

Colin

On the 6th of February I received another email:

Dear Colin

Sorry for not getting back earlier. We had sent the data to Oxford admissions people and they  disputed a minor detail which we now have changed. The data has been (or is being) sent to them again for an official review and comment before we go to the press. I've attached the figures on a powerpoint which I have asked for Oxford to review. Basically, I'm searching for dons who would like to comment on the figures. 

Let me know if you would be interested in commenting. I'll update you as to the Official response. 

Kurien

This had a powerpoint presentation attached to it consisting of  a bunch of bar graphs of admission success rates by subject, ethnic group and A level grade. These were quite interesting, but I was still unclear about the nature of the data so next day I asked for more information:

Dear Kurien

Thanks for this, very interesting. I assume the data come from official UCAS returns? I think there are too many "devil is in the details" type questions that I'd want answers to before I'd venture any substantive comment, basically I like to understand exactly how the numbers are produced, but I can suggest  something that would make your story have more impact here. Unless you believe the admission process is totally deterministic, admission depends, partly, on luck. Year on year the dice can roll in different ways. This in and of itself will generate apparent group differences. Even if you have what appears to be a census this will be true. Therefore you need to give the reader some sense of the likely variability in your numbers that is attributable to the non systematic part of the process that you believe is driving the differences you illustrate. In other words, some confidence intervals around your proportions are needed. I realize this is a big ask for a newspaper audience, so in lieu of that you could add the raw numbers to the bars, then the nit pickers can figure out for themselves the likely chance variability and focus on the ones where there are apparently real unexplained differences.

all best and good luck with your story

Colin

Later  the same afternoon I got this reply and another powerpoint presentation:

Hi Colin

I think you are basically wondering if the numbers are statistically valid. I don't think the trends seen are just a year on year variation due to chance. I should have put some numbers under the figures, so you could see there are sufficient numbers in the data to give it validity. The data is for three years from 2009-2011.

I'll ask that some numbers are put when the story is published. I didn't do a statistical validity test for all the plots. I did one for the medicine plot and got  a very low p number (less than 10^-12) and I've had it confirmed by someone else that p was less than 0.0001 for null hypothesis all applicants with the same grade has an equal chance of admission. The data actually was provided by the University. I've attached the slides with the numbers for the most important plots. 

The plots with Indian and Chinese applicants are only there to show even those groups with higher on average A level scores still underperform quite substantially in the interview. 

The only other caveat is that applicants for 2009 didn't have A*, which basically means the columns A*AA and higher grades are for applicants from 2010 and 2011, and in the plots applicants with AAA have relatively higher success rates (Oxford gave the data like that).

Thanks again

Kurien

And shortly afterwards I was sent an Excel spreadsheet with the raw numbers in it. On the 8th, after spending some time looking at the numbers I replied in the following vein:

Dear Kurien
Many thanks again.  I think I can give you a comment now, several in fact.
To make things concrete let's focus on Medicine. I attach the PDF of  a plot that  that I constructed from the numbers you kindly sent me. 

 

What it shows is that for 3 out of the 4 possible contrasts there  is evidence that the two ethnic groups differ in the offer rate. In the 4th (the A*AA group) the difference between the groups could plausibly be accounted for by random error. Put this way the evidence looks a little more modest than the impressively small p. values you calculate.  But, a reasonable person would, I think, begin to be persuaded that there is something worth looking into here. The question is, what?
Would, for instance, a reasonable person be justified in believing that graphs like this constituted strong evidence for an explanation in terms of discrimination - either conscious or unconscious? They might be tempted to: but they would be wrong. Now I'll tell you why I think that.
The main reason is very straightforward. These data don't actually tell us what we want to know. If I understand the information correctly the A level grades are (for the most part) those obtained after an offer has been made to a candidate. So at the point an offer is made the admission tutor doesn't actually know what grades the candidate has obtained. They will, of course, have other exam performance information about the candidates-  GCSE, AS levels and  predicted A2 grades  - and it is these, not the A2 grades candidates actually obtain that should be used in a like-for-like  comparison. To do otherwise is to put the admissions tutors in the dock for (apparently perversely) disregarding information that, at the time they made the decision, they could not possibly have known. To put it another way, candidates that look similar in terms of the A2 grades they achieve may actually look quite different in terms of the exam profile they presented at the time offers were made. Is this actually the case? I don't know, but I would want to know it before I leaped to the conclusion that admission tutors were doing something they should not be doing.
The second problem with these data is that they don't capture an important part of the admissions process - the candidates choice of college. About 80% of applicants make a specific college choice. Some colleges are much more popular than others ie Balliol and some much less popular ie St Hilda's. If you apply to a very popular college your chances of being accepted by that college are, by definition, lower than if you apply to a less popular college. I'm told that serious efforts are made to pass promising candidates who fail to get into their chosen college on to other colleges for consideration. How successful those candidates eventually are I don't know, but their chances of admission can hardly be enhanced by being passed around. For better or worse Oxford is a collegiate university and the influence of the colleges, especially at u/g level is very strong. To cut a long story short, it is essential to know and include in your data analysis information about first choice college so that you can rule out the possibility that all, or part, of the difference is simply a consequence of ethnic minority candidates being more ambitious in their college choices.
If you can eliminate both of these factors as possible explanations, you have a smoking gun and attention could fairly be focused on what else goes on in the rest of the admission process that might be disadvantaging one group or another. I don't want to comment specifically about what goes on in medicine for the simple fact that I don't know anything about it. Closer to home in the social sciences my own preference would be for admission without an entrance interview only after the A2 grades are known  and  a lottery amongst those who have passed a threshold level of grade acceptability. But it is easy for me to say that as I don't  have to reorganize the UK's daft  UCAS system or indeed give undergraduate tutorials. I suspect this is far from the mainstream view amongst undergraduate tutors, but to find out you will have to ask them.
I don't know if any of this will be useful to you, but it was interesting.
all best
Colin

Later in the day I got a reply:

Hi Colin

Thanks for looking at the data and for you comments! I think the confidence level goes down when one looks at each individual grade set for an individual subject because the sample size becomes very small. What is most striking is how the same pattern of success rates is seen in different subjects and in the overall data. 

Regarding your other comments, the following is my personal view (not the newspaper's, etc.):

I personally would disagree. I think figures suggest a major problem, for whatever reason- even if conscious or unconscious bias does not exist- which I think is unlikely-, the process as a whole definitely discriminates, at least in terms of outcome, in favour, and very substantially, of the majority race. The end result is a white applicant with lower grades has a higher chance of admission than an ethnic minority applicant. Frankly I think that is a real shame and one the university ought to be actively addressing. 

I understand the difficulty admissions tutors have without A level grades at time of decision making. However it would be worthwhile noting, 1., A level results  correlate really well with admission success. Having 3 A*s as opposed to AAA seem to increase ones chances by a factor 3 to 5 fold.

 2. A high performing student at A level, it is reasonable to believe, will have prior grades and recommendations that reflect that. If the prior information doesn't, then one wonders why admissions tutors use it. I can see an attempt could be made for the buck could be passed  to schools for under-predicting A level scores. 

Also, I don't think the choice of college is supposed to (officially anyway) make a difference in the success rate. If it does the university either should take steps to eliminate it, or change its official position. Also the idea that ethnic minorities are more likely to apply for the more competitive colleges lacks evidence, and in my mind credibility.

If anecdotal evidence is anything to go by, ethnic minorities, in Cambridge anyway, weren't that numerous in the more prestigious colleges, and it was the so called poorer colleges that celebrated multicultural diversity and an 'international feel'. I had a friend who was in Merton who told me he only saw one black student there in the 4 years he spent at Oxford. I'll leave it for the university to conduct an investigation. 

For years Oxford and Cambridge claimed ethnic minorities have lower success rates because they apply to more competitive courses. Neither ever felt the need to release the data to substantiate that claim- which is why I asked for it. Now that explanation has been debunked, I suppose,  like the  mythical hydra's neck, new unsubstantiated explanations will arise-  the general presuposition being any explanation is more likely than the possibility that dons or the system have a racial bias, and a theoretically unsurmountable burden of proof needs to be overcome before recognition that a bias is granted.  

Fact is racial bias is well documented in this country. Ethnic minorities are the 'first to be fired, and the last to be hired' in  the job market. They are disproportionately over-qualified and earn on average far less. Ofsted reported ethnic minorities are undermarked in university modules by up to 12 percent when anonymous marking isn't used . Chinese graduates earn, in spite of better qualifications, 25% less than white peers. I could go on. 

Either way, in my opinion, interviews should be scrapped. I don't trust, especially given the figures, the dons to be impartial. I think there should be anonymously marked - quite difficult- entrance exams. I think a lottery will harm the prestige of the university and I can't see how it could differentiate between the top performing candidates. Alternatively admission could be purely based on A level module marks. Cambridge has been going on about for sometime how A level module marks (as opposed to grade divisions) are the best predictors of degree performance. 

Anyway, these are just my personal opinions, and not at all related to the hopefully impending news story. Thanks getting back to me! 

Best wishes

Kurien

What I get from this is a strong sense of a conclusion in search of some evidence. OK, fine, it's journalism after all and that's the difference between advocacy and scientific research. I think the danger is though that strong beliefs can get in the way of diagnosing where the problem is and even if there is a problem at all. I'll post again with some more thoughts on what needs to be done to figure out whether there is actually a story here.

How did we get here?

When I look at the content of  most of the British sociology journals I feel like I've landed on an alien planet (some of the American ones make me feel like an alien too, but usually in a different way). Either I don't understand what it is that  people claim to be studying and writing about or I do understand it and it is so banal or absurd that you would be ashamed to share it with an intelligent  human being. James Davis' evaluation of the situation, which is almost 20 years old, gets to the heart of the matter (though I don't agree with absolutely everything he says).

On the other hand let's try to stay positive and give three cheers for  Gary King who has put  a lot of effort effort into thinking about how we can  use social scientific knowledge and modern communications technology to improve how we teach. You can watch him on video here giving a talk to the Harvard Board of Overseers. What he is advocating is so obviously the way we should go I don't understand why we aren't doing it already.

Occupations and Earnings Inequality

A very belated plug for a blog entry by my former student Mark Williams, now at the LSE. He has done some fascinating work on occupations and the growth in earnings inequality. What he finds is that, in the UK at least, growing earnings inequality is predominantly a matter of the occupational averages growing further apart as opposed to growing heterogeneity within occupational groups. So much then for the neoclassical soup, that bitter acid that was going to dissolve all that nasty  rent seeking activity that lay behind the sociological categories of occupation and class.

RIP Colin Tipton

I've just learned from the Guardian's obituary page that my old Surrey colleague Colin Tipton passed away in December after a long illness. He was one of that fast disappearing breed of academics in the social sciences who believed that teaching was an important part of the job, not something to be shuffled off to grad students or an inconvenience that got in the way of travel to foreign conferences. His course on Social and Economic history was always packed and students were inspired both by the depth of his knowledge and by his entertaining  approach to lecturing. Hidden behind his casual style was a deep understanding of how to communicate enthusiasm and love of a subject to  undergraduates some of whom were, to say the least,  ambivalent about studying. As his obituary makes clear he was  a delightful conversationalist and a committed member of the Socialist Workers Party, two attributes that aren't always found together, especially in Guildford. 

I still have fond memories of giving him a lift  to Oxford where he was  lecturing to some obscure left-wing group or other. His very funny stories  made the weary miles fly past.  My impression was that his sincere commitment did not prevent him viewing his political life with a degree of gentle ironic detachment that endeared him to everyone he came into contact with.