The Difficulty With Data

The Difficulty With Data Over the last few weeks Ive posted entries about diversity vs merit and the holistic admissions process. And while I hope that these entries have contributed some insight into how and why we do the things we do, one complaint in the comments on those entries was about a lack of data to accompany and support the claims I had made. As one commenter put it: MIT should release the full set of admissions data stripped of personally identifying information and let the community analyze it, because in the scientific community we trust data and analysis, not assertions. So lets discuss admissions data. First, Id like to say that Im a huge fan of statistics. I read 538 and Football Outsiders every day. When it comes to baseball Im a converted sabremetrician. In the natural world, I believe in the scientific method, which is to say I believe in data-driven analyses of phenomena, empirical evidence, and testable hypothesis as the best, and sometimes only, route to understanding most things which occur in our universe. But there is a problem with social science, and that problem is this: sometimes, you dont have all of the data, either because it is unavailable to you, or because something cant be captured. And then, if you try to build a model based on these incomplete data, you are liable to draw conclusions consistent with the data but descriptively incorrect. At its most basic form, its a variant of  post hoc ergo propter hoc after this, therefore because of this. The rooster crows, then the sun rises; all hail the befeathered Sun King! In more complex forms, its a very subtle misattribution of traits based on the ontologies used to characterize them, which begets an epistomelogical crisis: what do we measure and how do we measure it? Is the trait thus measured determinative or merely descriptive? And so forth. But lets back away from the analytical theory for a moment and ground what Im saying in some concrete examples. Heres another comment from my diversity vs merit post about SAT scores: This pretty much sums it up: SAT Math 750-800 15% 700-740 10% 650-690 5% From what you wrote youd think being in the 700-740 range and being in the 750-800 range doesnt have much impact on your chance of admission, but theres a 50% difference. Now, I and others are on the record as saying that we admit people, not test scores, and that in any case there is really not a difference in our process between someone who scores, say, a 740 on the SAT math, and someone who scores an 800 on the SAT math. So why, as the commentor asks, is there such a difference in the admit rate? Aha! Clearly we DO prefer higher SAT scores! Well no, we dont. What we prefer are things which may coincide with higher SAT scores. For example, a student who receives a gold medal at the IMO is probably more likely to score an 800 on the math SAT than a 740. But if we take an IMO medalist (with an 800) over random applicant X (with a 740), does that mean we preferred an 800 to a 740? No. It means we preferred the IMO medalist, who also happened to get an 800! The same goes for people who are highly ranked in their graduating class. Almost half of the class of 2015 were valedictorians of their high school. Aha! MIT must highly value class rank in our application!  No, we dont. Then why does this happen? Because we do highly value certain academic accomplishments, and if you are doing well enough academically to achieve these things, then you are probably doing pretty well in high school. Additionally, we highly value strong letters of recommendation, and often teachers strongly support students who really blow them away academically. So we select for these other traits and end up, as a side effect, with a disproportionate number of valedictorians. But its not because theyre valedictorians that we select them, but rather that because of the things for which we select they are valedictorians. Or, to  paraphrase a line from Llewellyn: being a valedictorian isnt the reason for the decision; its the result of factors which were reason for the decision. You see what happens here. Its correlation misdiagnosed as causation, and then interpreted through a particular narrative frame to conform (and confirm) to prior expectations. This  happens all the time in shoddy social science. And it inevitably occurs with whatever data we do release. If we released admit rate by state, it would be:  The admit rate for students from Wisconsin went up 2%, MIT must really want applicants from Wisconsin! When the reality would be much closer to: we took whom we wanted to take, and they were from Wisconsin. Was Wisconsin considered in a complex ecology of decisionmaking? To some degree, yes; thats what we mean when we say we read everything and have a contextual, holistic process. But was it a  determinative characteristic, one which could be separated out as a causal agent? Could Wisconsin be assigned a standard weight in a model of our decision process? Absolutely not. Whats happening here is a fundamental confusion between our admissions process and the results of that process. When we say that the admit rate for students with a 750-800 was 15%, it does not mean that the chances of a given applicant who scores between 750-800 if 15%. It means that those students whom we chose to admit included 15% of those who scored within the 750-800 range. Its a subtle distinction, but an important one in understanding the agency of admissions. Think of it as the difference between a living thing and its fossil. A fossil isnt the plant or animal itself: its the mineral imprint of the stuff thats left behind. Or think of it like a shadow. A shadow is not the thing which casts a shadow. Its the contours of where the light isnt. Thats how our admissions data work. It shows you where the decision wasnt. It shows you the shape of our decisions, not the basis on which they were made. Admissions data are an accretion of the the sediment which dropped to the bottom of the decisions delta, and not the moving river where the actual action happened. But Jurassic Park was a work of fiction, and just like you cant reanimate a velociraptor from its fossil, you cant understand the life of an applicant from the shadow of their data. This is why I hate chance threads so much. When an applicant says I have X SAT score and Y GPA, what are my chances to get into MIT its not a question I or anyone else can answer. Because, within certain bounds of sufficient academic preparation, the decision isnt made on these easily extracted and quantified points of data. The decision is about everything else. The response to this, of course, is well, so release the data on everything else! To which I ask: how? How can we meaningfully quantify how much a teacher supports a student? How can we meaningfully quantify that particularly poignant essay which shows a students resolve, or that particularly funny essay that makes us love their personality? Even if we did construct, ex nihilo, categorical cubbies to shove these interactions and experiences into, isnt that the same subjectivity wearing an objective mask? I dont think that Rate this applicants leadership from 1 to 5 is a particularly objective exercise just because we slapped a number in it. Trying to convert inherently subjective interpretations to objective quantities is like wearing fashionable glasses of an incorrect prescription: it may look hip, but all it ultimately does is cloud your vision. I understand that for the initial commentor and others this may be an unsatisfying explanation. MIT is a community which loves data, where people believe data can do anything, and where any explanation which undercuts the utility of data seems suspiciously unscientific. But Clay Shirky once gave a talk about how memes jokes, YouTube videos, lolcats, whatever spread through the Internet, and he said something to the effect that the physics of memes were more like the physics of weather than the physics of a falling object. We understand how things fall pretty well, and we can be pretty accurate in our understanding of when and where and how fast it will drop. But even though we have reams and reams of data about the weather, because of its utter complexity the best way to characterize what will happen the next day is often no better than partially cloudy with a chance of rain. Well, the physics of holistic admissions are akin to Shirkys idea of social weather. Based on easily apprehendable information, you might know roughly what the temperature (of an applicant) will be, and hazard a guess as to whether it will rain. But until all of the ingredients mix together in our admissions committees, like a storm forming over the gulf, you dont know upon whom a ray of sun will break through the clouds until it actually, finally happens.