If the SAT were a joke, you wouldn't expect scores to follow a bell curve with only ~0.3% of people getting a perfect score, but they do.

This isn't a good argument, because the SAT (and other tests of the same kind) are normalised to have this property.

The SAT may not be good tests, but that fact has nothing to do with the presence or absence of a normal distribution.

The SATs have a top score, if it were easy to game you'd find many people would clump at the top. That would make it impossible to normalize that distribution into a bell-shape because they're all clustered into the same bucket with no way of spreading them.

OP's point was that the perfect scores remain consistently outside of people's grasp despite the variety of resources available to prep for the exam. I only once managed to hit perfect score and my other best scores were one or two questions off. I had been taking the test since I was 11 (for various extracurricular camps/activities) and prepped multiple times for them. The biggest scores jumps were more closely related with my age and academic achievements than anything else.

That is correct.

So the SATs are (almost certainly) 3-pl (actually 2) IRT models. Essentially, it's a multivariate generalised linear mixed model to estimate both question difficulty and participant ability.

Normally, they'll estimate the abilities on the logistic scale, and use the percentile to back transform to a standard normal.

Most people don't cluster at the top because they are a good proxy for g, which is an imaginary statistical construct that we use to explain differences in school outcomes.

So I had a long digression here about the usefulness of penalties for guessing, but it turns out the SATs don't do that anymore, so wth?

(ETS invented IRT, that's why I'm pretty sure).

A lot of the reason the SATs are able to maintain a spread that lets them normalize the distribution is that they fill the tests with stupid tricks that fool people into wasting time. If you do a ton of prep then you learn to spot these tricks and then a lot of the questions become really easy and you can knock them off quickly.

A poor smart student may have mastery of the material but their score will suffer if they don't know the tricks.

And you can find out about all these tricks for free on Khan Academy or for $20 in a SAT prep book.

Not everyone can afford to sink time into learning SAT tricks.

No, but I think the point is a combination of Chesterton fence and Goodhart's law. The personal statments and the like, by becoming the primary targets, will cease to measure achivement, and instead measure wealth. So instead of changing without a plan, make a plan to replace SAT2 with something less subject to wealth

MIT is removing a test which rewards the wealthy and punishes the poor. That's good. But the other metrics that MIT currently use are more punishing to the poor, and more rewarding to the wealthy.

With the removal of this element, MIT is left relying on personal statements, recommendations, and the like. With those metrics as the targets, the kids of the wealthy will fund their child's various activities for the sake of giving them a better application.

I find this truly baffling. How can it be considered good use of time for someone to sink endless hours of time into learning the ins and outs of some stupid test that is not useful for anything? Just think about the man years of enthusiasm and creativity wasted on learning to score on a test, instead of something truly useful.

I understand why the students do it, I don't understand why anyone would consider it a good system.

There are plenty of kids in the US who can't afford to bring lunch to school, let alone pay $20 for a SAT prep book. Many of these kids don't have a computer or Internet access at home for that matter, so Khan Academy isn't much good to them either. When we talk about rich vs poor kids, there's a HUGE gap that many people don't realize.

Anecdote: When I was younger, I used to take a public bus to a local bookstore and work through the SAT practice books (on separate scrap paper of course).

It sounds like you didn't have to pick up younger siblings at school when you were younger. That's the reality for a lot of poor kids. Raised by a single parent, the older siblings are in charge of looking after the younger ones. The parent often does not get home from work until late (and may have multiple jobs).

I did actually. I was the oldest of three.

I'm not saying the poor aren't disadvantaged. It's just that the disadvantages are more along the lines of attitudes towards education in the first place (along with everything else). There are also fewer people nearby that serve as an "example" on how to learn or get ahead.

I don’t think that gap is huge in terms of internet access - there’s certainly a gap but it’s mostly a function of parents limiting access in my experience.

84% of teens have smartphones in the US. A large percentage of that remaining 16% have access to internet at school, as 98% of schools have broadband internet.

They have public libraries with both computers and books. Access has never been the issue if there is the will.

> The SATs have a top score, if it were easy to game you'd find many people would clump at the top.

I understand and appreciate my school was statistically abnormal but that is exactly what happened. I graduated in a class 334 students and several achieved perfect scores. Perhaps more than a third of the students achieved greater than 1300 and greater than 20% achieved greater than 1400. This is when 1600 was the perfect score.

All these scores indicate is the degree of conditioning imposed upon a student. I know people want these tests to mean something more for personal reasons, but according to all available data this is biased wishful thinking. The research on standardized convergent testing indicates it is not a measure of academic success or potential, but rather an indicator/discriminator of class distinctions due to availability of preparation.

Because I did not come from well groomed pedigree, did not value the subculture of excess vanity, and came from a family that was lower positioned financially I deliberately inverted the goals for a personal social experiment. I wanted to see how close to the bottom I could get and still graduate on time. This was exciting because the risks were greater. If you fail to estimate the conditions correctly you don’t graduate whereas other people get a slightly lower test score or grade point average. Because the goals were different you had plan and weigh the conditions in unexpected ways. Even with all the effort I put in there were still 5 people who graduated with a lower class rank than me.

What impact did that have in later life? I did not get a free ride to an Ivy League school like many of my classmates, enter corporate management immediately out of college, or become a corporate executive within 10 years. I did enter and graduate college. I became a self taught software developer without much challenge and have found very low resistance attaining employment as a senior developer in my full time job. I also became a managing principal in my part time job without as much effort. The greatest tragedy in all this isn’t lost status or earning potential but how boring life has turned out.

If you had a multiple choice quiz with the actual questions hidden you would get a normal distribution in results. Anything where you sum lots of small random effects.

If the test was not measuring anything at all, then you would expect repeat takers to have completely unrelated scores on first and last test.

That's true, I'm just pointing out that the normality of the distribution of the results alone proves nothing.

You're just bragging that you hit the ceiling on those tests.

That doesn't mean tests don't work; if you wanted to, you could have found much harder ones. (That's what I did, out of necessity. They wouldn't have given my app a second look if I hadn't.)

A lot of people would argue that Mensa itself is a joke...

That's irrelevant, what matters is that a high IQ society has a vested interest in measuring IQ as a proxy for general intelligence and standardized testing results should reliably correlate to IQ and general intelligence.

But is IQ really a proxy for general intelligence?

Yes. https://en.wikipedia.org/wiki/G_factor_(psychometrics)

An individual's performance on one type of cognitive task tends to be comparable to that person's performance on other kinds of cognitive tasks.

It's not impossible that these tests measure something other than general intelligence, that just happens to correlate with social outcomes (like historical, relative familial wealth).

IIRC general intelligence also doesn't simply measure speed of cognition, but also ability to choose what to focus on ("intuition"), which these tests do not measure.

Or intelligence is correlated with familial wealth because it is hereditary and smarter people tend to end up in more complex cognitively demanding professions which tend to pay more.

Maybe? The confounding factors are many. How to account for war (which often accrues wealth to the most brutal among the reasonably intelligent), disease (where selection is based on behavior, e.g., adherence to cultural norms, not necessarily intelligence), disparities in wealth across geography and culture (relative shifts in the normal distribution of IQ across demographics are reflected exactly in measures of wealth)? Let alone that procreation necessarily requires the joining of individuals who might have disparities in wealth, or intelligence, or both.

I tend to err on the side of caution with this one.

Yes, and it's the best proxy we have.

This is more of a joke story, but a few years ago a friend of mine who is neither ambitious nor succesful surprisingly scored 165-170 on their test. When they offered him a membership and told him the membership fee, his reaction was - Alright, but as I am in the top 5% of your members - should not you be paying me to hang out with you? :D :D :D

I always tell people this who want "only high IQ" people to run society.

"Oh, you want the people at Mensa to run your society?"

Well to be fair, those are the subset of high IQ people who have the time/desire to join a club about having a high IQ. Kind of an adverse selection bias

The overwhelming majority of the high IQ are not Mensa members. They are academics, doctors, lawyers, successful businesspeople, the kinds of people for whom it is more common in their friends group to have a Master’s degree than to be a high school dropout.

Some people consider a person with a Master's degree to be a dropout. (yes, I know, not necessarily...)

Those people are called academics. The kinds of Master’s degrees that are rent seeking credentialism, MBAs, MSWs, M.Ed.s are far more common than consolation prizes for those who didn’t get their Ph.D.

Perhaps, but the context I heard it in was self-deprecating from someone working in industry, relative to engineers with PhDs.