Statistical Chicanery

My lovely wife received the following via email from a relative who has never been comfortable with (or quiet about) the fact that we homeschooled:

WASHINGTON, July 14 — The Education Department reported on Friday that children in public schools generally performed as well or better in reading and mathematics than comparable children in private schools. The exception was in eighth-grade reading, where the private school counterparts fared better.

The report, which compared fourth- and eighth-grade reading and math scores in 2003 from nearly 7,000 public schools and more than 530 private schools, found that fourth graders attending public school did significantly better in math than comparable fourth graders in private schools. Additionally, it found that students in conservative Christian schools lagged significantly behind their counterparts in public schools on eighth-grade math.

This study has been all over the blogosphere, especially on the port side where anything the government does is by definition superior to what individuals do. And it's an especially good dig when government schools outperform "conservative Christian" ones.

But any time liberals use statistics to prove what you know isn't true, the first thing to do is look for weasel words. And here they are: "comparable children." What does that mean? I didn't know, so I did something relatives and liberal bloggers can't be bothered to do: I read the report. And I discovered that it doesn't support the conclusion that relatives and liberal bloggers draw - that public school kids perform just as well as private school kids.

The report at first gets a bit redundant:

In the first set of analyses, all private schools were compared to all public schools. The average private school mean reading score was 14.7 points higher than the average public school mean reading score...

In the first set of analyses, all private schools were again compared to all public schools. The average private school mean mathematics score was 7.8 points higher than the average public school mean mathematics score...

In the first set of analyses, all private schools were compared to all public schools. The average private school mean reading score was 18.1 points higher than the average public school mean reading score...

In the first set of analyses, all private schools were again compared to all public schools. The average private school mean mathematics score was 12.3 points higher than the average public school mean mathematics score...

And it gets redundant because in every single category private schools outperformed public ones, just like everyone with a room temperature IQ already knew would be the case. In fact, the higher the grade, the bigger the difference in performance.

So how can one look at those facts and conclude there is no difference? Leave the liars and damned liars and head straight for statistics, of course. In this case "hierarchical linear modeling," which is a fancy way to adjust student scores for gender, race, ethnicity, disabilities, eligibility for reduced-price school lunches (no kidding), computers and books in the home, Title I, and my personal favourite: number of absences. Think of it as Affirmative Action for people who can't be bothered to go to class.

Then they adjusted scores by school for aggregate student absences, teacher experience and certification, school size, ethnic makeup, and a host of other factors, because a big warehouse school full of kids who never go to class can't be compared to a small school where children are expected to attend. That wouldn't be fair. So don't change the school, adjust the numbers and the school will come out just fine.

Once they adjusted out every aggregate difference between students and schools they concluded that schools were all the same. Brilliant, except for one thing: the differences are important in the real world. Private schools still outperform public schools in every measured category, which is why people who want the best for their kids send them to private school or school them themselves. They want their kids in small schools where they are expected to attend and are more likely to excel.

But since we're playing with statistics, I found one that matters:

Nationwide, public school teachers are almost twice as likely as other parents to choose private schools for their own children, the study by the Thomas B. Fordham Institute found.

'nuff said.