Newsweek has once again issued its list of America’s Best High Schools. They’re using the same goofy formula as before: the number of students from a school who show up for AP or IB exams, divided by the number who graduate. Just showing up for an exam raises your school’s rating; graduating lowers your school’s rating.
As before, my hypothetical Monkey High is still the best high school in the universe. Monkey High has a strict admissions policy, allowing only monkeys to enroll. The monkeys are required to attend AP and IB exams; but they learn nothing and thus fail to graduate. Monkey High has an infinite rating on Newsweek’s scale.
Also as before, Newsweek excludes selective schools whose students have high SAT scores. Several such schools appear on a special list, with the mind-bending caption “Newsweek excluded these high performers from the list of America’s Best High Schools because so many of their students score well above the average on the SAT and ACT.” Some of these schools were relegated to the same list last year – and still, they’re not even trying to lower their SAT scores!
Newsweek’s FAQ tries to defend the formula, but actually only argues that it’s good for more students to take challenging courses. True, but that’s not what Newsweek measures. They also quote some studies, which don’t support their formula [emphasis added]:
Studies by U.S. Department of Education senior researcher Clifford Adelman in 1999 and 2005 showed that the best predictors of college graduation were not good high-school grades or test scores, but whether or not a student had an intense academic experience in high school. Such experiences were produced by taking higher-level math and English courses and struggling with the demands of college-level courses like AP or IB. Two recent studies looked at more than 150,000 students in California and Texas and found if they had passing scores on AP exams they were more likely to do well academically in college.
Worst of all, if parents pay attention to the Newsweek rankings, schools will have an incentive to maximize their scores, which they can do in three ways: (1) force more students to show up for AP/IB exams, whether or not they are academically prepared, (2) avoid having high SAT scores, (3) lower the school’s graduation rate, or at least don’t try too hard to raise it.
When asked why they publishing rankings at all, the FAQ’s answer includes this:
I am mildly ashamed of my reason for ranking, but I do it anyway. I want people to pay attention to this issue, because I think it is vitally important for the improvement of American high schools. Like most journalists, I learned long ago that we are tribal primates with a deep commitment to pecking orders.
As Monkey High principal, I agree wholeheartedly.
“The number of students from a school who show up for AP or IB exams, divided by the number who graduate.” … That’s insane. One of your fabled monkey students could come up with a better formula than that. So could I, and I nearly failed high school Algebra II. Let’s see, how about:
(% of enrolled students achieving passing AP test scores) x (% of students who graduate) = rating
Eliminates the advantages from encouraging unprepared students to take tests, prevents advantages from discouraging any students from taking tests, and makes it suicidal to prevent students from graduating. It even makes a nice, neat and readable scoring scale – the theoretical top rating is 100; the bottom of the barrel is 0 (which will happen if either term hits 0, like Monkey High’s grad rate). The most obvious problem with this rating method is that it can be manipulated by exclusive, elitist admission practices. A school that pre-selects only genius-level students has obvious advantages in the numbers competition. But as a 5-minute solution, it beats the stuffing out the arbitrary and nonsensical method that was actually used.
Actually, infinity can be treated as a number fairly consistently. (Some things won’t work, but without it we’re in the position where some things won’t work with zero, so…)
You don’t prove that if (x – 1) > 2 (x – 1)^2 > x; we have
(x – 1)^2 – x = x^2 – 3x + 1
It has roots at 3/2 +/- sqrt(5)/2; the parabola opens upward (the coefficient of x^2 is positive) so it’s greater than zero below the smaller root and above the larger root. I assume we’re only interested in the latter, which is about 2.618 (in fact it’s the golden mean plus one). For integers, you’re right; 3 and above (integers > 2) are smaller than the square of the next smaller integer, but there are non-integers x bigger than 2 for which (x – 1)^2 inf, because the usual behavior of quadratics in the plane doesn’t naively extrapolate to the Riemann sphere, or in the real numbers to the “Riemann circle”. Infinity plus or minus anything, or multiplied by anything other than zero, is clearly infinity. (Try doing some math with fractions, where one is the fraction 1/0, ignoring the oddity of that zero and following the usual rules to get a common denominator. You’ll find that it always ends up as something over zero, and it’s only 0/0 if you multiplied by zero somewhere, or subtracted from itself. But if you express infinity minus infinity as 1/0 and 2/0 you don’t see that problem; then only when you multiply it by zero.)
So when x is infinity, (x – 1)^2 is infinity.
(a/b – c/d = (ad – bc)/bd so 1/0 – 1/1 = (1 – 0)/0 = 1/0, squared is 1^2/0^1 or 1/0)
Which is not greater than x.
You’re right that there can be some paradoxes. The case of inf – inf is a case in point, not to mention inf/inf or (equivalently) times zero. That’s really no worse than having any number / 0 being problematical. In the field of complex analysis it’s routine to include a point at infinity as a member of the complex numbers, and to regard them as forming a sphere, the Riemann sphere mentioned earlier, which is remarkably well behaved mathematically.
Newsweek had to exclude schools with high average SAT, so the final list could have the kind of “diversity” that would make Newsweek editors feel good.
Actually, the fictional monkey high wouldn’t make the list, as dividing by zero is undefined, not infinity. (Infinity isn’t a number, but a concept. Suppose infinity has the property of being greater than all other numbers and
x = Inf
y = x-1
and
y > 2
then,
y**2 > x
which would be a violation of the initial conditions. I’m not 100% sure of the proof’s validity, but I am 100% sure that infinity isn’t a number, despite the fact some computer programs can deal with it as if it were.) So, while your Monkey University represents a good point, it would need at least /one/ successful graduate.
Actually, you have to let exactly one monkey graduate each year to avoid a division by zero exception.
More seriously, the Newsweek index, which is actively encouraged by the College Board, is not the only craziness coming out of CEEB there days. Starting in the fall, the College Board has declared that high schools will not be allowed to refer to courses as “advanced placement” unless the teacher has submitted a questionnaire and had it approved by CEEB. The reasons for this given on the AP Central Web site (http://apcentral.collegeboard.com/apc/public/courses/teachers_corner/51262.html) are patently phony: There was no clamor for this “audit” from coleges or secondary schools, the College Board itself publishes descriptions of what AP course content should be, and the results of AP exams tell colleges how to intepret the courses. Instead, this appears to be an effort by the College Board to protect its “AP” and “Advanced Placement” registered trademarks at the expense of teachers, who have better things to do than more paperwork. But the high schools are so intimidated by the College Board that there has been vritually no beyond grumbling by teachers.
It is possible for perverse correlations to occur, e.g. heavier vehicles being more fuel efficient because the lighter ones are driven by boy racers, or entrepreneurs with more failures being more successful than those with fewer.
It is possible that the better schools tend to overreach themselves, whereas the underachieving ones tend to be unable afford to waste entrance fees.
Then again, maybe people don’t want to know which is the most effective school, but which is patronised by the most affluent?
Maybe a school that has parents so rich they can waste a fortune on recalcitrant kids despite them having no chance of graduating is truly a gem of schools, that any parent would give their left arm to let their kids attend in order to mix with other well heeled kids?
If there was a community of plutocratic but frustrated parents with adopted pet chimps who could afford to have their prodigies attempt exams slightly beyond their intellect, then perhaps that Monkey High would indeed merit top ranking?
However, Monkey High must remain hypothetical because monkeys with such affluent guardians would educate them at home.
Someone could have plugged all the data into some genetic algorithm thingy and asked for the simplest possible metric that fitted received wisdom as to subjective ranking, and this metric popped out – despite actually being perverse and apparently easy to game.
Any metric used to reward or punish will become meaningless over time. Better metrics will survive for longer, worse ones for shorter periods. But given an incentive, humans will always overcome measurements. This is because a measurement is generally a proxy for some other real-world thing (eg, “school quality”) not easily specified or measured. The correlation between proxy and reality depends on expensive methods that would reduce that correlations not being employed. When the value of a better score on the measurement increases beyond the cost of the proxy-defeating method, the measurement loses accuracy. As more and more such methods become cost-effective, the proxy measurement falls into meaninglessness.
We see this on Wall Street all the time — investors focus on revenue, companies find ways to churn cash flows; investors focus on meeting targets, companies find ways to spike the targets; investors focus on ROIC, companies find ways to creatively shed capital into subsidiaries.
We see this in computer hardware — customers learn to check megahertz, pretty soon you have “performance-rated” MHz, clock step-downs for slow unts rather than step-ups for fast ones, chips sold clocked to the point of self-immolation (remember Cyrix?)
A measurement can only work in the long term if nobody cares much about it; if people care, it will diverge rapidly for whatever real phenomenon it was meant to proxy for. So long as nobody takes the Newsweek ranking seriously, nobody will found Monkey Hight; as soon as the Newsweek ranking matters, expect to see recruiting flyers passed out at the zoo the next week.
I see that metric author Jay Mathews is with the Washington Post. I also note that the top of the list is heavily populated with DC suburb (Arlington and Fairfax VA, Montgomery MD) schools. I suspect his emphasis on test taking might reflect a trend/bias in his area schools.
Yes, chosing your metrics is very important, and chod=sing bad metrics leads to bed conclusions, and ultimately bad practices.
An interesting example is the measurement of enrgy consumption in buildings, which all model energy codes do on a square foot basis, which may seem reasonable.
However, imagine that I design a very efficient building, that uses less sq feet thanaother design, and let’s just say this building is a laboratory or a hospital, both very intensive buildings.
The number of occupants (researchers/patients) are the primary drivers (all other things being equal, like the acuity of the patients or the demand of researchers for fume hoods, which just for the sake of this example we’ll say these factors are constant)
The big inefficient building could met code, while smaller more efficient building could have more trouble meeting code.
Of course, the standards of the code are not that difficult to meet, so this is that important, right now.
Of course the economy comes to the rescue–the more efficient building costs less to build and to operate, but maybe a little more expensive to design.
To be fair, they score very well in Poo Flinging 101.
The author of the list seems to take the schools that score well on the SAT and ACT as “all the smart kids go there for some reason I don’t understand” rather than “they help make their kids smart, and are academically competitive through admissions.” If all of these competitive schools are excluded, what are we left with?
Say the purpose of rankings is (ostensibly) to allow people to make decisions about where they want to go, and then attempt to get into these schools. Why bother, then, with a list of schools that “are good but not too good”? Are parents going to go out of their way to get their kids into these high schools? Is it possible for people to get into a distant high school, or are most students simply assigned to them by area (with the exception, of course, of the academically competitive who go to the unranked schools)?
At monkey high, we thump our chests the hardest.