November 28, 2015
Susan Dynarski had an interesting piece in the NYT on the relative effectiveness of charter schools in inner city and suburban neighborhoods. She reported on the findings from her own work, as well as others, that charter schools tend to result in higher achievement levels for inner city children, but had no effect on outcomes for children in suburban areas.
While this finding is interesting, it is important to note an important limitation to much of the research that has been done. Dynarski describes the nature of the tests:
“In the case of charter schools, researchers have found an innovative way to overcome selection bias: analyzing the admission lotteries that charters are required to run when they have more applicants than seats.
“Each lottery serves as a randomized trial, the gold standard of research methods. Random assignment lets us compare apples to apples: Lottery winners and losers are identical, on average, when they apply. Any differences that emerge after the lottery can safely be attributed to charter attendance.”
Actually the claim that differences in outcomes, “can safely be attributed to charter attendance,” is not true. There are two differences between the students who win the lottery and attend a charter school. One is the issue being examined, that they are attending a charter school. The other is that they are being placed in a school where the other students all have parents who were sufficiently motivated to enter their children in a lottery to try to get them in a better school.
It is reasonable to believe that the second effect will be very important. These parents clearly have more time and energy to get involved in their children’s education. This means, for example, if a teacher is not doing a good job or classroom conditions are not conducive to learning, they will likely complain to a school principal. All the students in a school will benefit from having such parents overseeing their kids’ education. Any school in which all the children had parents that selected the school because they considered it to be better for their kids would likely have better outcomes than a random neighborhood school which enrolled all children regardless of the interest of the parents in their kids’ education.
Just comparing the performance of lottery winners with lottery losers will be picking up both the parent effect and the school effect. A true randomized experiment testing the impact of the charter school itself would compare the test results of students who won the lottery and went to a charter school with a situation where children who won the lottery all went to a traditional school filled with other lottery winners.
It is possible that Dynarski and other researchers have a mechanism for controlling for this parent effect (I am not familiar with the details of this research — I welcome help), but the test as described in this piece is not a true test of the effectiveness of charter schools in improving education.
Correction: Susan Dynarski pointed me to a study she co-authored which compared test outcomes in charter schools with pilot schools. The latter also required parents to apply and were oversubscribed. The students in the pilot schools did no better than the students who lost the lotteries and were not able to attend the pilot schools. This should deal with the issue raised above of charter schools having students with parents more involved in their education than neighborhood schools. The pilot schools would also benefit from this effect, insofar as it is important.
Comments