School quality measures fail the test

Ranking the quality of a school's staff and administration by the number of Advanced Placement (AP) and International Baccalaureate (IB) tests students have taken is like evaluating the skill of a soldier based on the number of rounds he has fired.

Unfortunately, Washington Post Education columnist Jay Mathews's Challenge Index, used annually by Newsweek to rate high schools nationwide, does just this, dividing the total number of AP and IB exams taken at a school by the number of graduating seniors. Blair, with its wide selection of AP courses, strong teachers and special academic programs, finished only 32nd among area schools in last year's Challenge Index. The index is a poor indicator of staff quality because it fails to include any measurement of student success.

Two months ago, a Walter Johnson AP Statistics class wrote Mathews, proposing to change the formula to the total number of AP and IB tests passed divided by the size of the incoming freshman class. Changing the denominator to the size of the incoming freshman class was intended to respond to criticism that Mathews's formula favored schools with high dropout rates. While well-intentioned, the Walter Johnson students may not realize that there is no reason to punish schools with higher dropout rates, since student populations at such schools are, generally, less likely to do well on AP and IB tests than schools in more wealthy areas, like Walter Johnson.

The new formula also fails to take into consideration that wealthy students attend better-funded schools and have more access to AP and IB review materials, leaving poorer students at a distinct disadvantage in preparing for the exams. Mathews aptly identified the Walter Johnson students' solution as "effectively a measure of parental income." Their formula also rewards schools that encourage only top students to take AP and IB tests — only a minor improvement on Mathews's original version, which rewards schools that encourage less-prepared students to take AP and IB tests.

In order to improve their standing on Mathews's Challenge Index, schools in Charlotte-Mecklenburg County, North Carolina began offering to pay for students' AP tests. As a result, every county school made Mathews's list, which he claims contains the top five percent of high schools in the country. While the data on how many students passed AP and IB exams in Charlotte-Mecklenburg County is not available, only four out of the 19 county high schools met 2005 Adequate Yearly Progress (AYP) requirements in accordance with the No Child Left Behind Act.

Although AYP is not a perfect measure of a school's performance, the discrepancy between the school's failure to meet nationally-mandated standards and their inclusion on the Newsweek list shows that the Challenge Index is seriously flawed. Instead of recognizing that many students are not prepared for AP and IB courses, Mathews's index commends schools like those in Charlotte-Mecklenburg.

Mathews cites schools like Garfield High in East Los Angeles, where he says many low-income students are improving their AP scores, as proof that encouraging all types of students to take AP and IB tests improves their academic performance. But until Mathews presents data — and not just anecdotes — proving that his theory of increasing the number of AP and IB tests given actually improves the passing rates, his argument that schools with high Challenge Index scores offer the highest-quality administrators and teachers is not justified.

Indeed, a study conducted by the University of California, Berkeley's Center for Studies in Higher Education found that high scores on AP and IB examinations, not merely taking them, predicted college success. The Challenge Index may identify schools with high AP and IB participation, but declaring it an unbiased assessor of schools' staffs and administrations is misguided.

The best solution is to consider both the number of students taking AP and IB exams and the number of students passing, and then account for the number of students on Free and Reduced-price Meals, a measure correlated with family income. Instead of just including a separate statistic reporting the percentage of students at a school who passed at least one AP exam, as Mathews does, this revised formula would actually factor in the number of students passing an AP or IB exam. Doing so judges a school's staff and administration not by uncontrollable external factors, like the quality of students' previous education and home life, but rather by their efforts in the classroom.

Mathews claims that the Challenge Index's strength lies in its simplicity. What he and other Challenge Index proponents must understand is that, in dealing with so complicated a task as ranking the country's high schools, simplicity becomes a weakness.