Oxford's Institutional Bias

Time for the hair shirts. It's official, (or at least in the Guardian) Oxford appears to discriminate against undergraduate applicants from certain sorts of ethnic backgrounds. It so happens that I have some background information on this story which I'll let you into.

Before I tell my tale, let me make my position clear: 1) I don't know whether there is discrimination; 2) The evidence that has been put in the public domain is interesting, worth discussing, but very far from compelling; 3) I'm not a cheer leader for the Oxford admissions system; 4) I'm a believer in post A level grade admission to all universities, with, wherever feasible, grade threshold criteria (ie a driving test type set up in which everyone who meets the standard is eligible) for admission and a lottery in the event of over subscription. Before you tell me this is utopia, you should know that this has been and for all I know still is the way that the Netherlands handles entry to medical school. So, no interviews and no special tests unless there is an unarguable case for them, say on public interest grounds - ie given the restrictive practices of the BMA we might want to give a sleep deprivation tests to potential medics.

OK, so my story begins on 31st of January when I get the following email out of the blue from somebody called Kurien Parel:

Dr Mr Mills

I am researching for a  Telegraph story regarding what I believe to be a serious ethnic minority bias seen from Oxbridge Admissions statistics. I have found out that ethnic minorities with the same grades are significantly less likely (statistically) to be given an offer to study at Oxford after the interview process. The figures, in my opinion, are quite appalling. In medicine for example a white applicant is around twice as likely to be given an offer as a non-white applicant with the same grade. Indeed in medicine, and in other subjects, white applicants with fewer A*s in A level have significantly higher chance of admission than  non-white applicants.  

I want to know whether you would be interested in talking with me, potentially, to provide your perspective.  I can show you if you are interested some of the actual figures.  

Thanks

Kurien

Odd story for the Telegraph to be running with  is my first thought, so first I check this guy out. Googling the name brings up some hits for a  someone who seems to have done a post-graduate degree at Cambridge and a bit of student journalism for Cambridge's Varsity Online and not much else. Ah well, freelancers have to start somewhere.

I reply in an honest but bland way:

Dear Kurien

I'm afraid I'm not at all involved in undergraduate admissions (Nuffield is a graduate college) so don't have anything  enlightening or knowledge based to say. You would be better off talking to tutors in the u/g colleges who are directly involved in u/g admissions. They are the people that make the decisions that count.

I would be interested to know though what new data you have on Oxbridge admissions. I don't normally comment on anything unless I can examine the primary evidence myself.

all best with your story

Colin

On the 6th of February I received another email:

Dear Colin

Sorry for not getting back earlier. We had sent the data to Oxford admissions people and they  disputed a minor detail which we now have changed. The data has been (or is being) sent to them again for an official review and comment before we go to the press. I've attached the figures on a powerpoint which I have asked for Oxford to review. Basically, I'm searching for dons who would like to comment on the figures. 

Let me know if you would be interested in commenting. I'll update you as to the Official response. 

Kurien

This had a powerpoint presentation attached to it consisting of  a bunch of bar graphs of admission success rates by subject, ethnic group and A level grade. These were quite interesting, but I was still unclear about the nature of the data so next day I asked for more information:

Dear Kurien

Thanks for this, very interesting. I assume the data come from official UCAS returns? I think there are too many "devil is in the details" type questions that I'd want answers to before I'd venture any substantive comment, basically I like to understand exactly how the numbers are produced, but I can suggest  something that would make your story have more impact here. Unless you believe the admission process is totally deterministic, admission depends, partly, on luck. Year on year the dice can roll in different ways. This in and of itself will generate apparent group differences. Even if you have what appears to be a census this will be true. Therefore you need to give the reader some sense of the likely variability in your numbers that is attributable to the non systematic part of the process that you believe is driving the differences you illustrate. In other words, some confidence intervals around your proportions are needed. I realize this is a big ask for a newspaper audience, so in lieu of that you could add the raw numbers to the bars, then the nit pickers can figure out for themselves the likely chance variability and focus on the ones where there are apparently real unexplained differences.

all best and good luck with your story

Colin

Later  the same afternoon I got this reply and another powerpoint presentation:

Hi Colin

I think you are basically wondering if the numbers are statistically valid. I don't think the trends seen are just a year on year variation due to chance. I should have put some numbers under the figures, so you could see there are sufficient numbers in the data to give it validity. The data is for three years from 2009-2011.

I'll ask that some numbers are put when the story is published. I didn't do a statistical validity test for all the plots. I did one for the medicine plot and got  a very low p number (less than 10^-12) and I've had it confirmed by someone else that p was less than 0.0001 for null hypothesis all applicants with the same grade has an equal chance of admission. The data actually was provided by the University. I've attached the slides with the numbers for the most important plots. 

The plots with Indian and Chinese applicants are only there to show even those groups with higher on average A level scores still underperform quite substantially in the interview. 

The only other caveat is that applicants for 2009 didn't have A*, which basically means the columns A*AA and higher grades are for applicants from 2010 and 2011, and in the plots applicants with AAA have relatively higher success rates (Oxford gave the data like that).

Thanks again

Kurien

And shortly afterwards I was sent an Excel spreadsheet with the raw numbers in it. On the 8th, after spending some time looking at the numbers I replied in the following vein:

Dear Kurien
Many thanks again.  I think I can give you a comment now, several in fact.
To make things concrete let's focus on Medicine. I attach the PDF of  a plot that  that I constructed from the numbers you kindly sent me. 

 

What it shows is that for 3 out of the 4 possible contrasts there  is evidence that the two ethnic groups differ in the offer rate. In the 4th (the A*AA group) the difference between the groups could plausibly be accounted for by random error. Put this way the evidence looks a little more modest than the impressively small p. values you calculate.  But, a reasonable person would, I think, begin to be persuaded that there is something worth looking into here. The question is, what?
Would, for instance, a reasonable person be justified in believing that graphs like this constituted strong evidence for an explanation in terms of discrimination - either conscious or unconscious? They might be tempted to: but they would be wrong. Now I'll tell you why I think that.
The main reason is very straightforward. These data don't actually tell us what we want to know. If I understand the information correctly the A level grades are (for the most part) those obtained after an offer has been made to a candidate. So at the point an offer is made the admission tutor doesn't actually know what grades the candidate has obtained. They will, of course, have other exam performance information about the candidates-  GCSE, AS levels and  predicted A2 grades  - and it is these, not the A2 grades candidates actually obtain that should be used in a like-for-like  comparison. To do otherwise is to put the admissions tutors in the dock for (apparently perversely) disregarding information that, at the time they made the decision, they could not possibly have known. To put it another way, candidates that look similar in terms of the A2 grades they achieve may actually look quite different in terms of the exam profile they presented at the time offers were made. Is this actually the case? I don't know, but I would want to know it before I leaped to the conclusion that admission tutors were doing something they should not be doing.
The second problem with these data is that they don't capture an important part of the admissions process - the candidates choice of college. About 80% of applicants make a specific college choice. Some colleges are much more popular than others ie Balliol and some much less popular ie St Hilda's. If you apply to a very popular college your chances of being accepted by that college are, by definition, lower than if you apply to a less popular college. I'm told that serious efforts are made to pass promising candidates who fail to get into their chosen college on to other colleges for consideration. How successful those candidates eventually are I don't know, but their chances of admission can hardly be enhanced by being passed around. For better or worse Oxford is a collegiate university and the influence of the colleges, especially at u/g level is very strong. To cut a long story short, it is essential to know and include in your data analysis information about first choice college so that you can rule out the possibility that all, or part, of the difference is simply a consequence of ethnic minority candidates being more ambitious in their college choices.
If you can eliminate both of these factors as possible explanations, you have a smoking gun and attention could fairly be focused on what else goes on in the rest of the admission process that might be disadvantaging one group or another. I don't want to comment specifically about what goes on in medicine for the simple fact that I don't know anything about it. Closer to home in the social sciences my own preference would be for admission without an entrance interview only after the A2 grades are known  and  a lottery amongst those who have passed a threshold level of grade acceptability. But it is easy for me to say that as I don't  have to reorganize the UK's daft  UCAS system or indeed give undergraduate tutorials. I suspect this is far from the mainstream view amongst undergraduate tutors, but to find out you will have to ask them.
I don't know if any of this will be useful to you, but it was interesting.
all best
Colin

Later in the day I got a reply:

Hi Colin

Thanks for looking at the data and for you comments! I think the confidence level goes down when one looks at each individual grade set for an individual subject because the sample size becomes very small. What is most striking is how the same pattern of success rates is seen in different subjects and in the overall data. 

Regarding your other comments, the following is my personal view (not the newspaper's, etc.):

I personally would disagree. I think figures suggest a major problem, for whatever reason- even if conscious or unconscious bias does not exist- which I think is unlikely-, the process as a whole definitely discriminates, at least in terms of outcome, in favour, and very substantially, of the majority race. The end result is a white applicant with lower grades has a higher chance of admission than an ethnic minority applicant. Frankly I think that is a real shame and one the university ought to be actively addressing. 

I understand the difficulty admissions tutors have without A level grades at time of decision making. However it would be worthwhile noting, 1., A level results  correlate really well with admission success. Having 3 A*s as opposed to AAA seem to increase ones chances by a factor 3 to 5 fold.

 2. A high performing student at A level, it is reasonable to believe, will have prior grades and recommendations that reflect that. If the prior information doesn't, then one wonders why admissions tutors use it. I can see an attempt could be made for the buck could be passed  to schools for under-predicting A level scores. 

Also, I don't think the choice of college is supposed to (officially anyway) make a difference in the success rate. If it does the university either should take steps to eliminate it, or change its official position. Also the idea that ethnic minorities are more likely to apply for the more competitive colleges lacks evidence, and in my mind credibility.

If anecdotal evidence is anything to go by, ethnic minorities, in Cambridge anyway, weren't that numerous in the more prestigious colleges, and it was the so called poorer colleges that celebrated multicultural diversity and an 'international feel'. I had a friend who was in Merton who told me he only saw one black student there in the 4 years he spent at Oxford. I'll leave it for the university to conduct an investigation. 

For years Oxford and Cambridge claimed ethnic minorities have lower success rates because they apply to more competitive courses. Neither ever felt the need to release the data to substantiate that claim- which is why I asked for it. Now that explanation has been debunked, I suppose,  like the  mythical hydra's neck, new unsubstantiated explanations will arise-  the general presuposition being any explanation is more likely than the possibility that dons or the system have a racial bias, and a theoretically unsurmountable burden of proof needs to be overcome before recognition that a bias is granted.  

Fact is racial bias is well documented in this country. Ethnic minorities are the 'first to be fired, and the last to be hired' in  the job market. They are disproportionately over-qualified and earn on average far less. Ofsted reported ethnic minorities are undermarked in university modules by up to 12 percent when anonymous marking isn't used . Chinese graduates earn, in spite of better qualifications, 25% less than white peers. I could go on. 

Either way, in my opinion, interviews should be scrapped. I don't trust, especially given the figures, the dons to be impartial. I think there should be anonymously marked - quite difficult- entrance exams. I think a lottery will harm the prestige of the university and I can't see how it could differentiate between the top performing candidates. Alternatively admission could be purely based on A level module marks. Cambridge has been going on about for sometime how A level module marks (as opposed to grade divisions) are the best predictors of degree performance. 

Anyway, these are just my personal opinions, and not at all related to the hopefully impending news story. Thanks getting back to me! 

Best wishes

Kurien

What I get from this is a strong sense of a conclusion in search of some evidence. OK, fine, it's journalism after all and that's the difference between advocacy and scientific research. I think the danger is though that strong beliefs can get in the way of diagnosing where the problem is and even if there is a problem at all. I'll post again with some more thoughts on what needs to be done to figure out whether there is actually a story here.