How are the SAT/ACT not objective tests? Are they not objective measures of how you perform on the SAT/ACT? There are abundant low cost practice guides and study materials. I came from a very poor family and with some practice at home scored very well on the SAT. It was one of the few tools I had to attract college admissions boards. I feel like taking that away will severely disadvantage poor kids who are otherwise bright with strong work ethics.
There is some truth to the argument that standardized tests are a flawed proxy for the combination of intelligence and motivation to learn that one wants in a student, and that the test may be biased towards some cultures that have a tradition of preparing their children for those standardized tests, whether directly or indirectly. Despite the relative low cost of practice, it is not free. Also, each attempt of the test has a cost.
Still, I'm a fan of standardized tests. If there's a problem, we should try to improve the tests rather than doing away with them.
I mean, the SAT has "aptitude test" in the name; the idea that you can practice it and do better each time is a bug and not a feature.
But I think the bigger danger is that after a while, practicing for standardized tests starts monopolizing the curriculum at secondary schools, and teachers start being evaluated by their students' test scores to the exclusion of all else. Even 20 years ago, I remember getting herded onto a bus to some community college to take an extra "practice state proficiency test" whose score didn't count for anything at all.
Even if universities have lousy motivations, de-emphasizing that kind of thing probably makes high-schoolers' lives a little better.
Since 1993 SAT no longer stands for 'scholastic aptitude test'. It had its roots as a sort of IQ proxy but that has fallen out of favor because it doesn't reflect the full purpose of schooling (e.g. advancement from given means, value-add of schooling for a given student) nor range of things that cause students to achieve (e.g. hard work, conscientiousness, etc).
One could make a test called 'big brain test' and sell it to schools to say it measures big brains, and it might be only somewhat correlated but still get widespread adoption because it fulfills a need schools have to have some sort of quantitative sorting of students, but it still wouldn't necessarily be a measure of big brains. This is sort of what the history of the SAT is.
I'm an immigrant and moved here when I was 9. My parents always spoke in a different language growing up. I personally had a hard time with certain vocabulary that I never picked up and that maybe I otherwise would have if I had native English-speaking parents. This definitely slows you down when reading problems as you're only gathering about 3/4 of the problem at times. This is probably an outlier, but just thought I'd share my point of view here.
Statistically significant, such that it created some academic literature and enough cover for Harvard to make this new policy. Practically significant, maybe not.
Here's a comment from a different branch of this thread:
> The findings in general is that much of the disparity between minorities and whites goes away for harder analogies that are more technical or scientific in nature, in fact contrary to what I expected, the recommendation of the study is to simply eliminate the bottom third of verbal questions (ranked by difficulty) involving analogies or sentence completion, as those are the questions that are most likely to involve cultural bias.
First off, the SAT has gone through many revisions in the last 20 years so it's unclear this still applies. The SAT doesn't even have analogies any more, perhaps as a result of this research.
Secondly, that doesn't address differential prediction abilities. It addresses a concerning aspect of the verbal test, but is somewhat irrelevant when your goal is to predict college success. I'm not aware of differences on that metric.
Biased in the sense that cultures that that rewarded diligence and intelligence are able to compete more effectively against cultures that don't and the wages of these accomplishments can be seen in their posterity?
put a big IMO/IMU around all of this, but the SAT/ACT isn't objective in the sense that studying for them is something you can do to improve your score and your prospects. However, it is also transparent in how you do that: you take a course, get study books, and/or take practice tests and there are programs out that have been trying to help disadvantaged students get access to these tools. its also much more cost effective compared to the other solutions like going to a private school (difficult, can be costly without aid) or getting into expensive niche sports.
so imo getting rid of standardized tests does get rid of a subjective rating of an applicant which is good in theory, but when you leave in all of the other subjective ratings that are much more difficult for disadvantaged students to do well on it who is it actually benefiting?
subjective/objective are probably not the right adjectives here, but to use your example I would say that a timed foot race is an objective measure of how well you do in timed foot races, but is not an objective measure of your innate athletic ability because of training/coaching and you have to factor those to attempt to back into physical ability. the same is true for the SAT, in that people treat it like an intelligence/general knowledge test when you have to factor in the specific training that person has had for the SAT itself.
A timed foot race is a quite reasonable measurement of athletic capability - like, it's not a perfect one and a more thorough and diverse evaluation would be better, but if you don't have a better evaluation option then it's obvious that it has some information; if you'd take a hundred random people and split them into two halves based on foot race times, then that would be a far better separation than one based on randomness or any factors that measure athleticness even less.
I mean, it's okay to use the best imperfect metric you have available; so acknowledging "the metric is not fully correlated and depends on other factors" does not mean anything, the only argument that has weight is "hey, this other metric has better correlation with The True Thing and less bias, use that instead"; if the remaining alternative metrics are even less objective and more biased than SAT/ACT tests (which IMHO is the case), then removing SAT tests won't and can't improve things, despite these tests being flawed.
that would work if you were talking about random people who were selected to do a surprise track race, but if the track race is known to be THE measure and that can be influenced by training then you’re going to see skew.
to be clear, i still think the sat is better than everything else because preparation is much more legible and accessible compared to all of the other ways to stand out for college applications.
So what do you propose is more objective (or whatever qualifier you prefer) than the SAT?
Pointing out shortcomings of standardized tests is one thing. Advocating for eliminating what seems to be our most equal/standardized/objective/whatever measurement without a replacement seems like a bad idea to me.
if you read my first comment i never argued for getting rid of the sat, though they should get rid of the usurious prices for tests and sending your scores out. in fact, i argued that of all the subjective tests the sat has the most legible and accessible path to maximizing your score compared to all the other subjective things colleges look at.
That's an interesting read, and I personally am a fan of Scott's somewhat radical views on college education in America, but surely you recognize that you are directly side-stepping the question? The problem at hand here is not about "how to decide who to get into undergrad", it's "how to decide who to train for the 'elite' jobs in society".
I haunt thrift stores, and almost always there are SAT/ACT prep books for sale for a couple dollars. You can probably get them for free from your local library.
In my not-so-humble opinion, however, the best way to prep for the verbal SAT is to read books. Novels are fine.
For the math, prep by taking all the math classes your school offers, and make an effort to learn all the material.
Do that, and you won't need prep classes, and you'll also be well prepared for college.
While it's true that one can improve their standardized test scores by studying or taking a course, and the ability to do those things is correlated to a student's socio-economic situation; if you label standardized tests as subjective, then for me it follows that there is no objective measure of a student period.
Which begs the question, who exactly do you admit to your school, and why?
> I feel like taking that away will severely disadvantage poor kids who are otherwise bright with strong work ethics.
They can still take the test and get a high score.
> How are the SAT/ACT not objective tests? Are they not objective measures of how you perform on the SAT/ACT?
Most of the studies on this show that all of the cultural references in the reading passages and math word problems were things that wealthy people would have experienced in real life, but poor people would not, so it gave an advantage to wealthy kids. Also a lot of the vocab were words that wealthy kids had heard at home but not poor kids.
So while it was supposed to be a test of intelligence, it is really a test of how wealthy you were growing up.
This is pissing on people's leg and telling them its raining. There's no "cultural loading" in the SATs. English is my second language and I didn't even know Chicago was a big city until my mid-20s, but reading a lot and practicing vocabulary words is easy and cheap. New York's Stuyvesant high school has an average SAT score of around 1500, and half of students are eligible for free/reduced price lunch: https://www.nydailynews.com/opinion/stuyvesant-serves-needy-...
Poverty makes things harder, no doubt. But American parenting is just bad and American students don't try very hard. My wife comes from a lower-income white family (multi-generational lack of economic mobility) and the lack of emphasis on eduction or discipline at home is glaring. Some people, like my wife, are just naturally very self-driven and make it out. But a lot of potential gets wasted because kids are allowed to just faff around instead of being rigidly guided by parents (or, as is more commonly the case, by the one parent).
So then what do you make of the studies that show that SATs are culturally biased, like the one OP posted [1]. I decided to go ahead and buy it for the 10 bucks and review it. It's a surprisingly technical study that seems fairly robust and it has been replicated several times (original study in 2003 and replicated in 2010).
The findings are that much of the disparity between minorities and whites goes away for harder analogies that are more technical or scientific in nature, in fact contrary to what I expected, the recommendation of the study is to simply eliminate the bottom third of verbal questions (ranked by difficulty) involving analogies or sentence completion, as those are the questions that are most likely to involve cultural bias.
If my understanding is correct (and I'm sure I have some subtle details wrong), the idea is to let students take the SAT, then go over the verbal/analogy/sentence completion portion of the SAT and order each question by the proportion of white students who answered it correctly, and then eliminate the easiest of those questions from the final score. The most difficult questions are still part of the test, but apparently most of the cultural bias is among the easiest questions to answer that minorities get tripped up on.
If your hypothesis were true, that it's mostly a matter of bad parenting, it's kind of hard to see why bad parenting would make minorities struggle to answer the easiest questions for white people, but not also struggle to answer the hardest questions as well.
The racial gap is almost exactly as big on the SAT Math as it is on Verbal. Do you think SAT Math is also culturally biased? If so, do you think it is exactly as biased as Verbal? That is, why is the Math gap not, say, half the Verbal one, if the cause for disparities is cultural bias?
Why would there be a Black-white gap (two groups that come from the same American culture) but not an Asian-white gap (two groups that come from different cultures)?
Black and white families are not in fact part of the "same American culture". And, of course, it's not the case that Asians uniformly outperform white subjects on the SAT; "Asian" comprises many different cultures, and not all of them are empirically advantaged in America.
The problem is that the system is designed to identify the most effective parents, not the most talented children. If you look at college admission as a function of personal merit, then it's unfair to judge a child based on their parents' ability to raise them. If you look at it as a system to identify those who will benefit society most by being educated, then you're going to get a lot of false positives that have parents from higher socio-economic classes.
Providing a test that produces an empirical result is just making it very easy for parents to game the system. It's a single, stationary target that they can fire money at. If you take away the test you're still selecting for the best parenting (all of life is). But it lets universities put more emphasis on grades and personal stories, areas where students whose parents have never heard of the SAT can still excel.
If you can use money to game the test then it's culturally loaded toward people who have enough money to game the test. A broader admissions criteria doesn't have this specific flaw
There is no such thing as “personal merit” independent of parenting. Kids are a product of their genetics and upbringing.
And you’re missing the point re: income, which is that all poor kids aren’t similarly situated. Asian kids raised in the bottom 20% of the income scale have a 25% chance of ending up in the top 20% as adults. For white kids it’s only 11%. Socioeconomic status is a factor but it’s not dispositive or insurmountable.
As to grades—you think it’s easier for a bright but poor kid to do their homework in time for 12 years than to study for one test? As to “personal stories”—that’s just a vector for manipulation by the whites people who run the admissions system. It’s remarkable to me to read anecdotes from admissions counselors, who say stuff like that they are looking for weird and non-conforming kids: https://www.cnbc.com/2021/10/22/former-stanford-admissions-o.... Talk about cultural loading!
Even if you remove personal merit from the idea of college admissions, which most people do not, then you've still got a system that's selecting for the cultural knowledge and position of parents instead of the benefit received from educating a particular person. This seems to be working well enough so far, but it isn't optimal.
I never said that someone's income quantile was dispositive. If my parents never took the SAT and have no idea what it is, I'm at a huge disadvantage to do well on the test by any measure. Regardless of their income. Again, the SAT isn't testing for academic aptitude but for the cultural knowledge of parents and peers.
As a wealthy parent I know exactly how to throw money at prepping for one test: hire expensive tutors and purchase practice tests. Gaming years of GPA or coming up with a compelling story is more complicated and difficult to solve with the application of financial resources.
A system with vague and subjective criteria is subject to bias, for sure. They are also more difficult to manipulate directly with wealth (other than bribes). We have to decide what our priorities are there as a society.
With all due respect, the vast majority of us here took these tests. Do any of us remember them containing quizzes on obscure aspects of water polo and equestrianism and sailing jargon, or other things that only rich kids would know about? I remember them being fairly self-contained.
And that's just the reading comprehension. The idea that the math word problems involve cultural bias that would affect performance is utterly ridiculous. "Most studies"... there is no chance that any quality research has been done to support something so laughable.
I will look at any citations you may have, but I am 100% confident that you have been duped.
To be completely fair, I think at least some portions of the SAT are distinctly more difficult without a strong reading background.
You can definitely cram for the grammar / math portions, but the reading comprehension sort of just requires a lot of reading experience. There isn't a rote set of things you can memorize to get a high score, which could arguably be seen as good or bad.
Maybe more practically, we should consider: Do lower income students read less on average? And if so, is the difference greater than their ability to gain access to tutoring services / prep books which can be used for the crammable portions of the exam?
Succeeding in college also requires a strong reading background. The point of the test is to predict college success, not "be fair" (an inherently subjective statement)
This theory would make sense if the hardest questions showed the most racial disparity, but it's actually the easiest questions that do, according to the papers cited by jedberg.
Come on, man. Your opinions about the SAT are based on a single infamous SAT question (of unknown provenance) from many decades ago?
There are no questions like that any more. You really think the people designing the highest-profile standardized test in the country are oblivious to this kind of problem these days? There's a ton of cultural and racial bias screening and filtering that goes into producing these questions now, both manual and statistical. You can look at some sample questions yourself.
Being from Cincinnati, I had to laugh at that particular example a little bit, because there's an annual charity "Rubber Duck Regatta" where you pay to sponsor a rubber duck and they dump them all in the river and give a prize to whoever has theirs finish first. When I was a kid, they ran TV ads for it blasting the name out loud about every six seconds, so most students who didn't know the word otherwise would have figured it out. I'm sure there would be statistical analysts looking at the scores for that question by region and scratching their heads.
It doesn't say only wealthy people will know what it is, it says they are much more likely to know what it is because they are much more likely to have experienced it first hand.
It would be hard to deny that generally sailboat racing is done by wealthy people.
That doesn’t follow. Poor people are aware of many things they don’t do. You’re wanting to lower standards to help people who don’t need that kind of help to expiate you own status guilt.
Poor people may also not have cars. Or have ever flown on a plane or owned an iPhone. But they aren’t necessarily stupid and will probably know what these things are and if they happen to be then they should go to elite colleges.
OTOH, You could be wealthy but from Montana. You probably have never witnessed a regatta in your life. You and your wealthy friends will probably be able to answer every question related to a ski holiday however.
That appears to be a purely statistical argument - some questions appear to be disproportionately hard/easy for certain racial groups. That has some value, but it would be nice to see some concrete example(s) of how this bias is expressed.
Definitely my most hated section. Full of rare, obscure, you-know-it-or-don't words for massive bookworms.
But if you read the papers cited in the GGGP comment, those hard problems aren't the problems that show the racial gap. The hardest problems show the least gap. Which doesn't fit the pat little sailboat jargon theory at all. However it does fit a common pattern, also seen in longevity statistics, where the most exceptional are more similar to one another than the average are.
Sure! This idea is interesting and may well be original:
As people get older, the mortality rate gaps between different groups get smaller on a relative basis. Like, the ratio between rich and poor 50 year olds' risk of death over the next year might be 3x or more, but as they age, they become more similar, and by 90 the ratio may be more like 1.3x, and after 100 it becomes indistinguishable from 1 (in part because of thin data).
The theory is that the genetics and environmental conditions needed to survive to an old age are fairly specific, so once you're down to those people, they have less variation from one another. The health risk factors that distinguish income and racial groups may mostly impact younger age mortality, and the people that get past those younger ages may have avoided or resisted those risk factors, regardless of race or income.
Similarly here, students who "survive" to do well on hard SAT questions may be fundamentally very similar to one another (sharing a high g-factor), and this may greatly reduce socioeconomic and racial gaps that are seen on the easier questions.
Does this mean the easier questions are biased? Not necessarily, I would argue. The concepts tested in the easy-medium questions are most likely to be intellectually foundational -- you are far, FAR more likely to get obscure rich people jargon in the hard questions than the easy-medium ones. We want people to have the basic intellectual exposure that enables them to do well on those easier questions, and we should reward it.
The problem is not the test, the problem is racially biased failure to educate in the lead-up to the test. This explains the pattern perfectly: the strongest students power through regardless of background, while the weaker are more sensitive to resources they were given. So you see the biggest gaps on easy-medium questions where basic educational background matters most to getting them right, rather than raw talent.
I see. Very curious. Shall we call it “adverse convergence” hypothesis?
Have you explored broader implications of your idea? For example people who survived tribulations of a political career may end up close to each other despite their initial difference in political opinions, and maybe even across different countries. This doesn’t sound quite as convincing, but I’m kind of grasping in the dark trying to find some other examples.
There's probably a name for it somewhere! I want to call it something like... convergence of survivors, broadly construed. (Achievers? Overcomers?)
I don't know of any examples outside of health but health is a big enough topic and closely related enough to education that I think it's likely to have some application.
Certain psychometric studies also support higher sensitivity to environment in the lower economic strata. Not exactly the same but you see the connection.
The studies cited by jedberg are very interesting, I just don't think they show what he thinks they do.
A better alternative to trashing the SAT is to provide an estimated adversity score, as ETS recently began to do, so schools have a sense of what a student may have had to overcome to reach their level of achievement.
Yes, it's not trivial because evolution stops caring about you after you reproduce.
It's also a little unclear whether we should consider the % difference convergence remarkable, given that mortality at older ages is so much higher and the absolute difference may not show much convergence.
Aside from the fact, repeatedly pointed out, that the analogy section has been kaput since 2005... this would be a hard question, the sort that shows the least racial and income gaps.
No, it's a measure of how well you perform on the subjects tested by the SAT. There are many things that correlate with high SAT scores (e.g., grades, being Asian?), but it is definitely true that you can prep for the SAT without any meaningful amount of money. I don't believe that anybody has ever alleged that SAT scores are a proxy for intelligence, though I suspect there is a decent correlation. In my mind it is a useful, objective, universal measure of how an individual student is able to prepare for and execute on a difficult task.
