On March 25, The Atlanta Journal-Constitution published an unprecedented investigation that found suspicious school test scores across the United States. That story built on reporting we had been doing since 2008 on test scores in Atlanta and Georgia public schools.
After the AJC’s analysis of Atlanta scores led to a state investigation and 2011 findings of widespread cheating, a national testing expert suggested we could do the same thing on a nationwide scale. We spent the next seven months investigating.
The federal No Child Left Behind Act requires each state to give a statewide standardized test to all students in grades 3 through 8 to measure performance in reading and math. In Georgia, that is the Criterion-Referenced Competency Test.
A team of three reporters and two database specialists spent months collecting databases of standardized test scores in those grades for 69,000 schools, in 14,743 districts in 49 states. (The 50th, Nebraska, didn’t have usable data because it didn’t give a statewide standardized test until last year.)
Some states, including Texas and California, post online the data we needed. Most states sent data within days or even hours. A few were more challenging. We called state education departments and made formal open records requests. Some states required months of negotiating and multiple requests before they sent data.
With the data in hand, we used a method similar to the analysis used to find suspicious test scores in Atlanta. It compares test scores achieved by a “cohort” of students: That is, when a third-grade class in a school moves on to fourth grade, the group is likely to remain similar and so test scores won’t vary a lot.
By plotting large changes in scores for a cohort, for better or worse, an analyst can identify test results that are highly unlikely to happen by chance. When scores go up that much, it suggests some intervention, such as cheating, to change the expected results.
Scores that drop are meaningful, too. Test scores can rise or fall dramatically because one teacher cheats and the next one does not, or vice versa.
In addition, patterns in test scores may show dramatic declines, as they did in Atlanta, after cheating is exposed or investigated. In both cases, scores can drop because cheating stops.
Then, we did another level of analysis, gauging the likelihood that abnormal score changes would occur in “clusters” of grades in one district. We calculated the probability of districts achieving huge changes in test scores in a lot of classes, compared to the probability statewide. In some cases, the probability was less than one in 1 trillion.
We presented our findings by phone and email to officials in problematic districts for response and we talked to executives and testing specialists in those districts and states. When district officials raised concerns we couldn’t immediately answer, we went back to our data to check our results. In the meantime, we talked to national experts and decision makers on testing and education policy.
The Latest
Featured
