Two years ago, forensic auditor and education researcher Jarod Apperson listed reasons on this blog to be optimistic about the turnaround plans of Atlanta Public Schools ands its leader Meria Carstarphen. But he also noted it was rare nationally for superintendents and their initiatives to dramatically improve student achievement.
In that 2016 Get Schooled column, Apperson wrote:
The sad reality is that aside from marginal accomplishments superintendents can highlight, most leave districts largely in the same state they found them. This is not because most superintendents are bad but because the task is so daunting. A few years from now, we will know whether Carstarphen's strategy has managed to break that reliable mold.
After the release of the 2017-2018 Georgia Milestones scores Friday, Apperson says Atlanta now can be assessed by two years of results since its ambitious turnaround plan, which brought in charter groups to run some schools and beefed up academic supports at others.
Apperson has been providing statistical analyses of Atlanta schools for several years on his Grading Atlanta blog. Using the Milestones scores, he digs into the results to see whether Atlanta's ambitious experiment is paying off yet.
Overall, his response is “not really.”
If you are interested in Atlanta schools, check out his blog, which is a must-read for Atlanta parents and educators. A graduate of New York University in finance and accounting, Apperson earned a doctorate in economics this year from Georgia State University and teaches at Spelman College.
Apperson writes:
As I explained a few years ago, the score releases have so many numbers and are hard enough to interpret that they tend to provide lots of fodder for misguided newspaper headlines, tweets, and emoji high-fives. In general, the conversation overstates how much fundamentals in a district or school change from one year to the next. I've tried to take a relatively rigorous look at five questions that interest me about Atlanta Public Schools.
It's worth caveating that all the comments below are based on test scores. If you believe test scores are effective measures of what we want schools to teach students, you may find these conclusions useful. If not, nothing below will likely be of interest to you because this is about the Georgia Milestones.
In what readers might consider the most important question, Apperson examines whether achievement in APS has changed.
Apperson says:
"Not really. For each of the past seven years, APS has performed similarly relative to the state. Back in 2011, the average student in the district scored 0.62 grade levels behind the state. In 2018, the average student scored 0.55 grade levels behind the state. The highest score occurred in 2015 and the lowest in 2012.
One reason APS scores haven't moved much is that it's a large district. Small districts tend to see more variation in their scores from one year to the next. However, the stability in APS scores is noticeable even relative to similarly sized districts. Among large metro districts, Fulton, Gwinnett, Cobb, Clayton and Forsyth all saw score changes that were more substantial than those in APS (both up and down).
A second reason that APS scores haven't changed much is that the needs of students served by the district remain relatively stable. The district is slowly gentrifying. Over the past four years, APS has produced a consistent measure of needs for test takers in the district. That measure is called the challenge index and you can read more about it here. It fell slightly from 60 to 58 over the past four years.
Overall, performance in APS is about the same. It has perhaps ticked up very slightly, and its students have become slightly less needy, but overall the changes are incredibly modest.
Apperson asks whether APS achievement is more equitable, concluding:
No. Outcomes in APS are not more equitable today than they were a few years ago. Across the country, huge gaps exist between poor students and their wealthier peers. For the nation as a whole, students at the wealthiest schools score about 6 grade levels above students at the poorest schools.
Are the turnaround partnerships increasing achievement?
Possibly. APS launched a turnaround plan a few years ago. One component of the plan transfers management of six district schools to charter operators (Purpose Built, Kindezi, and KIPP). The partnerships are being phased in over several years. Results are available now to allow us to look at the 2016 and 2017 partnerships to see how they've done.
Apperson also examines a second plank in the APS reform plan, providing targeted resources to several schools that performed poorly in the past. The plan included small-group tutoring and mentoring of principals.
He concludes:
Overall, there is not much evidence that these investments are paying off. In 2017 about half the schools saw scores rise while the other half saw scores fall. In 2018, results look a little better, but that picture doesn't hold up once you control for changes in the challenge index. It seems like year-to-year changes at these schools may be driven by noise and school-specific things (for example leadership or staffing changes) rather than the targeted intervention strategy. The biggest increase among the targeted schools occurred at Barack & Michelle Obama Academy. The biggest decline occurred at Scott Elementary.
And, finally, Apperson examines a question of national interest with the focus on charter schools. He looks at whether the charter operators are beating the odds and finds:
KIPP and Drew are. Others are not. About 20 percent of APS K-8 students now attend a locally approved charter school. In 2018, students at KIPP scored a half a grade level above what would be expected of them based on the needs of the students who attend. Drew students also performed better than would be expected by 0.4 grade levels.
All other charter operators performed worse than would be expected based on the students who attend. The worst performance occurred at Atlanta Classical. Students at that school scored 1.2 grade levels behind what would be expected. In math skills, which are often thought to change more with school inputs, the performance was 1.8 grade levels behind. Even though Atlanta Classical has one of the wealthiest student populations in the state, about half the schools in the state score better at math. Students at the traditional public school, Jackson Elementary, have a similar challenge index. However, in math, students at Jackson score 2.5 grade levels ahead of Atlanta Classical.
