Fact-Checking Criticism for Charters in Atlanta

Critics say it’s unfair to compare charters to traditional public schools nearby, because they don’t serve the same students. Is that true?



The challenge:

Atlanta’s charter school programs were showing strong results for students, and the district wanted to learn from their successes. But ongoing criticism alleging that the charters’ success was due to their alleged selectivity undermined the opportunity to compare practices and learn from the programs.


The intervention:

SDP Fellows Gayle Burnett and Rubye Sullivan analyzed student and teacher populations in charter and traditional public schools and Atlanta to look for differences, and used results on a high-school reading exam to understand the impact of attending a charter school on student learning.



The impact:

There were few differences in enrollment at charters and traditional public schools, and the SDP fellows’ analysis found that the positive effect of attending a charter in middle school was roughly the same as having a significantly more effective teacher. They also found that fewer charter school students transferred compared to students at district schools, and that charter teachers had far less experience. This analysis inspired SDP fellows to later design a causal study in conjunction with their SDP Harvard faculty advisor to further isolate impacts.

The Challenge:

Fact-Checking Criticism for Charters

Charter schools in Atlanta were on track to serve 15 percent of city students, and many had emerged as leaders in improving student outcomes. The district, which oversees the charters, had in-depth performance data for them, and wanted to study their programs in order to replicate their success.

Enrollment works differently at charters compared to traditional public schools in Atlanta. Students from across the city could apply and win entry by lottery. Critics commonly allege that charters “cherry-pick” the students most likely to succeed in their programs, unlike traditional neighborhood schools, and “lemon-drop” students with particular challenges, such as those in need of special education or limited-English-proficiency programs, or students with poor academic performance, attendance, or behavior.

These criticisms had overtaken any discussion of performance at Atlanta charters. Were they especially effective, or did they just have better-prepared students?

The Intervention:

Comparing Students and Outcomes Across all APS Schools

SDP Fellows Gayle Burnett and Rubye Sullivan compared the population of charter and non-charter students in Atlanta, including students’ race and gender and whether they qualified for subsidized school meals, special-education services, or limited-English-proficiency support. They found that students were broadly similar, teachers were broadly different, and that the impact of attending a charter school was significant and positive for students.

Students at both types of schools were similar in terms of race and gender, though charters enrolled fewer Hispanic students than traditional district schools. They also found that fewer charter students were eligible for free- or reduced-price school meals—an effect they attributed to an overall more affluent student body at three out of 17 charter schools districtwide.

Burnett and Sullivan also looked to see whether charter school students were more likely to change schools than their counterparts elsewhere. By comparing “exit code” records, they found that students in traditional district schools were much more likely to have switched: 22 percent, compared to 7 percent of charter students.

To find differences among teachers, they compared groups using the best common data point available: years of experience. They found that non-charter teachers had 12.7 years of experience on average, compared to 4.6 years for charter teachers.

A separate analysis found a positive effect on reading outcomes for charter schools, based on scores on a statewide test on 2014 and a variety of statistical models that accounted for student and school characteristics. The most positive effect was in middle school, where attending a charter was roughly the same as having a highly effective teacher instead of a mediocre one.

The Impact:

Lessons Learned

SDP Fellows Burnett and Sullivan concluded that in Atlanta, the success of local charter schools is largely due to efforts within the schools themselves, not outside factors. In contrast to common criticism, they did not find “cherry-picking” or “lemon-dropping” to be factors in the schools’ success, because the student populations at charters and traditional public schools were similar and charter students were less, rather than more, likely to switch schools once they had enrolled.

These findings were designed to either confirm or dispel the myth that charter schools perform differently based on who they select to enroll. The SDP fellows planned to continue their analysis with additional lottery data on incoming charter students. These preliminary results were presented to support the exchange of ideas and collaboration between charter schools and traditional district schools.

SDP Resources:

Read Gayle and Rubye's SDP capstone report on exploring community perceptions of charter schools.