Should you vote for Opportunity School District? Depends on how much you trust state's charter school record.

In alerting me to his analysis, Apperson said, "I'm sure you are probably suffering from OSD fatigue at this point."

While I've been hearing about the OSD from advocates and opponents on the front lines, I haven't heard many "regular" Georgians raise the issue until this weekend when a half-dozen people asked me about it. I believe the blitz of pro and con TV commercials has increased awareness.  I even saw several yard signs around metro Atlanta.

With that, here is Apperson's commentary:

By Jarod Apperson

Depending on which ad you’ve seen, Gov. Nathan Deal’s Opportunity School District is either a white knight coming to save public education or a headless horseman coming to pillage the state’s most vulnerable communities.

Such simplistic appeals are inevitable when the general population is asked to vote on an issue that is complicated and requires a great deal of background knowledge to engage with substantively.

As someone with expertise in this area, I feel comfortable saying that frankly we don’t know how this endeavor might turn out if it is approved. There is a real possibility the OSD will improve education, there is a real possibility it will have little impact, and there is a real possibility it will do harm.  An informed vote for or against the OSD depends on which of those possibilities you think is most likely and the extent to which you believe the state should take a risk.  Below I give my take on several key questions and lay out the best available evidence.  I will leave it to readers to weigh the evidence, which points in different directions, and reach their own conclusions about the OSD’s prospects.

What will the OSD do?

The gist: Turn over the management of selected schools from the local school district’s central office to a charter operator selected by an appointee of the Governor.

The detail: Voters will approve or deny the OSD by voting on Amendment 1, appearing on ballots statewide with the following language:

Provides greater flexibility and state accountability to fix failing schools through increasing community involvement. Shall the Constitution of Georgia be amended to allow the state to intervene in chronically failing public schools in order to improve student performance?

Anyone with knowledge of the OSD will recognize this statement fails to paint a clear picture of what the initiative hopes to do.  Based on this description, one could be forgiven for believing Amendment 1 hoped to raise student achievement by encouraging more community bake sales.  That’s not the plan.  But the overly genial language alone doesn’t mean it is a bad idea.

The authorizing legislation spells out more clearly the tools the state will have at its disposal when intervening in schools. They include managing the school directly, stipulating changes the local school district must make, shutting the school down, and selecting a charter organization to operate the school.  It is clear the governor's preferred course is to select charter organizations to operate the schools, a model used in Louisiana and Tennessee, states that inspired the proposal.

Now that we are clear on what the OSD hopes to do, the most pressing question comes down to whether OSD-eligible schools will be better off or worse off managed by charter organizations. I’ll come back to that discussion in a moment, but first I want to talk a bit about the identification of OSD-eligible schools.

Does the OSD do a good job of identifying low-quality schools?

The gist: Sort of, but it more consistently picks up high-poverty schools than low-quality schools.

The detail: Each year, the state puts out a score it calls College and Career Ready Performance Index ("CCRPI"), which is mostly based on crunching standardized test scores different ways.  This metric forms the basis for schools being selected for the OSD. Any school that scores below a 60 for three years in a row becomes eligible.  One reasonable  critique of CCRPI is it doesn't do a very good job of comparing schools to their peers — other Georgia schools that serve similar students.

Instead, it systematically ranks schools with poor students low and schools with relatively rich students high.  In reality, there are low-quality and high-quality schools at all income levels. (See Apperson's interactive chart here that presents a better measure of school quality and poverty for all schools in the state.[1]  Highlighted schools are schools that rank in the bottom 6% (the share of Georgia schools that are OSD-eligible) of student growth relative to peer schools.

It is clear the variation in quality at the high-income end is just as wide as the variation at the low-income end. Because the OSD relies on CCRPI rather than the school quality measure presented above, schools deemed eligible tend to systematically be poor schools, rather than schools that have achieved the lowest academic gains relative to their peers. I have created the same chart of quality and poverty, but highlighted the OSD-eligible schools rather than the schools in the bottom 6%.

There is no school without at least 35% of students in poverty that qualifies for the OSD. In contrast, one out of every three schools with more than 60% of students in poverty is on the OSD list. So having a sizable share of poor students is essentially a prerequisite for being selected.  Still, the schools chosen do tend to be below average quality. They may not be the worst schools in the state (and some even have high growth!), but they average around the 28th percentile.

This discussion so far about school quality – as measured by both the CCRPI and my own approach – relies on test scores.  But we ultimately care about whether schools prepare students for successful lives, not whether they can score well on a bubble test at the end of third grade. That brings me to the next question.

Are standardized test scores good metrics for measuring school quality?

The gist: Generally yes, but not always.

The detail: Over the past five years, the relationship between test score gains and long run outcomes has been a topic of great academic interest.  The most compelling evidence suggests teachers and schools that are able to achieve high growth on test scores cause their students to succeed later in life. However, it is also possible for schools to raise test scores using means that do not impart the skills necessary for later success.

There have been three major papers presenting high-quality evidence that schools and/or teachers who are able to raise test scores ultimately cause their students to have better long run outcomes.  Chetty et al. (2014) shows that high-growth New York City teachers reduce teen childbearing, increase college going, and increase earnings at age 28.  Dobbie & Fryer (2015)shows that a high-scoring Harlem school reduces teen pregnancy and incarceration rates. Angrist et al. (2016) finds that Boston charters able to raise test scores also increase four-year college going.

A fourth study finds more mixed evidence.  Dobbie & Fryer (2016) analyze Texas charter schools.  They find schools that negatively affect test scores also negatively affect four-year college enrollment and earnings (consistent with findings from the studies above). However, in contrast to the other evidence, schools that are able to raise test scores do not improve long-run outcomes.  One possible explanation the authors provide is that the high-scoring schools in the study may have focused too narrowly on tested skills, taking time away from the development of non-tested skills important for long-run success.

Collectively, these papers suggest test scores are a good proxy for whether schools and teachers are imparting the skills students will need to succeed; however, they also suggest it is possible for schools to achieve high scores without developing those skills.

If test scores are a meaningful measure of skill development and OSD-eligible schools do not now succeed at raising test scores (recall that they on average rank at the 28th percentile in quality), the logical next question is should we expect the schools to do any better if they were taken over by the OSD. Since the governor’s preferred intervention is to select charter operators, the answer hinges on the quality of those operators.

What is the evidence on how charter schools currently operating in Georgia affect standardized test scores?

The gist: Local charters are slightly above average, state charters are significantly below average, and within both groups there is a great deal of variation from school to school.

The detail: Georgia now has about 60 start-up charter schools that operate in grades tested annually (grades 3-8 take Milestones End of Grade Tests).  Before they opened, those schools were reviewed and approved by either the local school board (“Local Charters”) or the State Charter School Commission (“State Charters”).

On average, the charter schools now operating in Georgia are lower quality than traditional public schools. Much like traditional schools, the quality varies a great deal.  Some of the best schools in the state are charters.  Some of the worst schools in the state are charters. Go here to see the same chart of school quality and poverty we looked at before, but now local and state charters are highlighted.

If the OSD could ensure the charter operators it partnered with would achieve results similar to the four KIPP schools (all are 98 or above on the quality measure, compared to 28 for the OSD schools), voting for the amendment would be a no brainer.  But that is probably optimistic to say the least. Most charter applicants don’t come with a proven track record, making it tough for authorizers to ensure quality at the time charters are approved.

If instead, the OSD were to partner with schools similar to the average state-approved charter, schools taken over would likely end up achieving at even lower levels than they are today (State charters’ average quality is 12, even lower than the 28 for OSD schools).  For me, this uncertainty about quality is what causes the most skepticism of Amendment 1’s prospects.

Will the OSD charter operators be like the shining examples of what is possible (KIPP) or will they be subpar (like the average state charter)?

There are some reasons to believe the OSD charter partners will be more successful than state-approved charter schools. First, the funding will be higher. State approved charters are funded at a rate lower than most nearby traditional public schools, and they have to spend part of their funding on facilities. The OSD will fund schools like locally approved charters and give them facilities. Second, the OSD will be tasked with seeking out high-quality charter operators. Depending on how savvy the OSD leader is, he or she may find partners with proven track records elsewhere in the country.

On the other hand, there are reasons to believe the OSD charter partners will be of similar quality to the state-approved charter schools (i.e. worse than the OSD schools themselves). First, there is a limited pool of people capable of starting a high-quality charter.

If anything can be learned from the gap between the results from locally approved charters and state charters, it is probably that good charters tend to get approved locally. It takes an incredible amount of time and dedication to run a successful charter school.

My sense is the size of the high-quality charter school community is more constrained by the number of leaders capable of developing and implementing a strong plan than it is by local districts unfairly rejecting great proposals. If that’s indeed the case, the OSD will likely struggle to find great operators. Those out there are already opening local charter schools.  Second, it appears the OSD may be biting off more than it can chew.

The proposal would allow the OSD to take over up to 20 schools a year (the agency could elect to take over fewer schools). The scope of that potential undertaking is striking given there are only about 20 good charter schools in the whole state today and it took almost two decades to get here. The notion the OSD could open 20 schools of good quality in a single year seems tenuous. I would feel more comfortable if the plan was two per year, rather than 20.

At the end of the day, I think the governor has good intentions and wants to see the OSD-eligible schools improve for the kids who attend them. I don’t buy the narrative he is looking to exploit children to profit his friends (though I do think there are organizations out there who would like to profit from the initiative). I also believe there is plenty of room for improvement at OSD schools.

But I am less confident the OSD will partner with charter organizations capable of delivering that improvement.

If Georgia had a history of holding its charter schools to a high standard, I would feel more comfortable supporting Amendment 1. But with the mixed reality that exists today, supporting the amendment would require me to trust Georgia will raise the charter quality bar in the future, partnering with high-quality organizations.

If that is a risk you are willing to take, vote yes.

If instead you believe the state needs to demonstrate more consistent results from the charters already operating before taking on a new initiative, vote no.

[1] This measure of school quality is the three-year average Student Growth Percentile, with controls for observable characteristics of the students at the school. School performance on this measure is then used to rank schools by percentile. Percentile ranks are helpful for intuitively discussing one school relative to others; however, they may overstate differences around the center of the distribution. Schools between the 40th and th 60th percentile in the state probably differ from each other in less dramatic ways than schools between the 80th and 100th percentile.  If you want to see more about how this is calculated, you can access the data and the STATA code here.