I have a confession to make: I love the U.S. News & World Report Best Colleges website. I'm a premium subscriber, with access to all of the details behind the controversial rankings. NAIS is officially and staunchly opposed to school ranking systems, and I support that position. But I also believe that good data can be highly valuable, to schools as well as families trying to do the very best for their children.
When our youngest was looking for a college, his main desire was for a school close to a warm beach with good waves. He is a good student, with wide-ranging academic and extracurricular interests, including a passion for surfing. He wanted a solid academic school, but much more. Through his own networking, he identified several colleges and universities, in the South, near good surfing beaches and with surfing programs. His first choice was a school I did not know much about.
We used the U.S. News website to pull down data that seemed particularly relevant to his prospective experience — the mix of in-state and out-of-state students (would the culture be too "local" for a kid from New Jersey), freshman retention rates (do kids tend to stick it out, or return home after a year), class sizes (he does better in small groups than lectures), and so forth. I was particularly concerned about freshman retention rates, and used the website to dig further. U.S. News linked me to the college accreditation report, where I could analyze freshman retention and graduation rates by home state and high school GPA: It turned out that kids from New Jersey with academic profiles like our son's tended to leave the school after one year and graduated at the scary rate of only 25 percent. He's not a data geek, like me, but he quickly decided to make a different choice. Today, he's thriving — and surfing — in his junior year, elsewhere.
The market has produced vastly better information systems about colleges than about K-12 schools. And, the situation just got worse. In recent weeks Niche, a website dedicated to helping consumers choose colleges, communities, and public schools, released its "Best Private High Schools" in America rankings. To be blunt, it is the least competent ranking system I have ever seen. I have heard concerns about the ratings from a number of our schools. I hope all of our schools will share my thoughts here with their communities. These are bad data.
The U.S. government does a pretty amazing job of trying to maintain an accurate count of the number of private schools in the United States. Every two years, the Department of Education combs state websites, association membership lists, online listings, even word-of-mouth details to obtain a complete list of private schools, from long-standing diocesan Catholic schools and traditional independent schools to fledgling schools of every stripe. The Department then mails every school a survey to ascertain information on mission, grade levels, enrollment, diversity, staffing — the operational essentials — and follows up with schools by mail and in person, until it obtains what it believes is a reliable picture of about 95 percent of all private schools. These data have been collected for 25 years, and are available in summary form and raw files. The Private School Universe Survey (PSS) provides the most comprehensive and accurate information on private schools, of all kinds.
The latest PSS identifies 30,861 private schools in the U.S. in 2012. Of that, 8,548 include 12th grades—qualifying them as high schools and evaluation by Niche. But being a private school recognized by the U.S. Department of Education does not make you eligible for consideration as one of the "Best Private High Schools in America." Niche grades fewer than half of all private schools (3,892) and includes slightly fewer than that (3,880) in its rankings. It's not because the un-ranked are weaker performers somehow; Niche simply lacks sufficient data on more than half of America's private schools and casually omits them from the rankings — noting their small sample sizes in a footnote.
In reality, Niche should have excluded more. Even the data on included schools are poor. Niche promises users of its new web tool that "a high ranking in Overall Experience generally indicates that: Students are very happy with their experiences in all aspects including academics, teachers, health, safety, resources, facilities, extracurriculars, sports, and fitness; the school is an exceptional academic institution in terms of teachers, students, resources for learning, and student outcomes; the school is made up of a diverse population and fosters an accepting, positive school culture; and students are actively involved in a variety of extracurriculars and sports the school offers." Not a bad concept of a good school. But the indicators are horrific.
These four virtues are measured by three "grades" — an Academics Grade, a Student Culture and Diversity Grade, and Survey Responses (capturing student, parent, and alumni satisfaction). What's behind these grades is what is most troubling. Now, I want to be clear about one other thing at this point: Niche's methodology summary is opaque. It is impossible for a researcher to validate exactly how the company did its calculations. That is Niche's right; it is a proprietary company, not a research organization. But consumers deserve to know how their schools are being judged. I'm using my best professional judgment to infer how Niche arrived at nearly 4,000 private school rankings.
There are no government sources for student outcomes at the school level for private schools. Most private schools do not administer state assessments, and the federal government does not collect SAT or ACT scores by school. Niche solves the problem by relying on student self-reports of SAT and ACT scores from a national survey of 16,688 students, parents, and alumni. This may sound like a large sample — but it's not. If every private high school in America were equally represented, each school would have two responses, and those might not even be students — with test scores. Niche recognizes this by requiring, arbitrarily, that a school have seven survey responses for data from the survey to be included in grading, for SAT/ACT scores and other measures of satisfaction.
Niche says explicitly that if its survey yielded seven survey responses, the school's student outcomes were estimated from that handful of responses. One does not have to be a professional pollster to know that the confidence intervals around such estimates are huge — actually, plus or minus 35 percent. The ranking system ignores confidence intervals. Schools that, in fact, differ by indistinguishable amounts statistically are lined up as if the differences matter.
In addition to survey estimates of test scores, Niche estimates academic quality with student-teacher ratios and four-year college matriculation rates, available through the PSS. At NAIS, we used the PSS data to analyze the student-teacher ratios and matriculation rates at every NAIS member school in a major private school market, New York City. The mean student-teacher ratio is 6.1 with a standard deviation of 2.0. We can say with 95 percent confidence that any New York City NAIS school will have a student-teacher ratio in the range of 2 to 10. In the bigger scheme of school performance, these are all low ratios with small differences among them. How the ratios affect academic programs depends on something not even measured — how staff are deployed. To the extent that these ratios serve as proxies for class size, research is very clear that differences of this magnitude are not significant. The same for college matriculation rates — the average NYC NAIS school sends 98 percent of its grads to college, with little variation across schools. There is zero validity in ranking schools that differ as little as ours on these measures. Yet that is exactly what Niche does.
The same crime is committed in the name of diversity. The federal PSS provides good data on student backgrounds. We used it to analyze diversity in the same NYC sample. The middle 50 percent of all NYC independent schools ranges from 15.7 to 29.6 percent students of color. That is a small band of scores for sorting a large number of schools. But that is what Niche does — often without any of the promised "accepting culture," available only if a school received seven survey responses. What virtue is there in a school that's 50 percent diverse, but students are isolated in racial cliques? Niche also gauges diversity with gender: The closer the student mix is to 50 percent, the higher the score. NYC co-ed independent high schools all hover around 50-50, but Niche downgrades schools that are 53-47: stupid.
Which brings me to the bottom line. The 16,688 response survey is what Niche touts as the differentiator of its methodology. It is what is supposed to take the rankings beyond crude objective statistics. Yet lots of ranked schools lack survey responses, eliminating reliable measures of satisfaction, as well as nuanced measures of academic quality and diversity. I urge readers to investigate the adequacy of data in your own communities. Here is a different major market example — Philadelphia. Of the top 10 schools in the Philly metro area as ranked by Niche, only one includes survey data. Among the top 25, only four schools have survey data. Poking around the website, you quickly realize the obvious about survey data: Small schools, like many independents, do not randomly generate many respondents in a national sample.
I write this critique recognizing that many NAIS schools top Niche's national and local lists. I do not want what I've shared here to detract from your well-derived reputations for excellence, earned through a wide range of good work for students. The Niche "Best Private High Schools" list does nothing to distinguish quality. Whether a school looks good or less good in this system, it would be best to help your community understand why the ranking just doesn't matter. NAIS is committed to providing our school community with better data. I believe that strong data systems can help all of us make far better decisions, as school leaders or parents. Niche is just not one of those systems.