Report on Atlanta school cheating inquiry validates AJC analysis

After The Atlanta Journal-Constitution began reporting evidence of possible cheating in Atlanta Public Schools, the school system commissioned an expert to challenge the findings. The results, issued in May, validated the AJC’s reporting, but the system never made the report public.

The report by University of Pennsylvania researchers replicated the AJC’s original test score analysis and confirmed the findings, with the benefit of additional data not available to the newspaper.

In the report, obtained this week by the AJC and Channel 2 Action News, researchers were asked to “identify alternative explanations” for the AJC’s findings.

But the report’s findings largely refute an explanation often offered by system officials, that high rates of student turnover at some schools could cause the statistically unlikely jumps in scores that suggest cheating on standardized tests. Even after the report was completed, Atlanta Superintendent Beverly Hall continued to say demographic shifts could explain some test score jumps.

The report is yet another red flag for the Atlanta school system’s embattled testing program. Over the last two years, Atlanta has faced a series of reports and investigations questioning improbable test score gains at some city schools. More than 100 city educators have been reported to the state teacher certification body and the state and the U.S. attorney’s office are now investigating.

“We’re pleased the report confirmed the accuracy of our reporting,” said Charles Gay, the AJC’s investigative team leader. “But we’re disappointed that officials decided to investigate our investigation rather than deal with the problem.”

The 13-page report, dated May 11, said that while dramatic swings in some Atlanta test results did not prove cheating occurred, the scores “do point to student achievement gains and losses that are highly unusual and for which cheating could be one explanation.”

System spokesman Keith Bromery said Thursday officials received the report in May and immediately turned it over to investigators who were looking into the allegations on behalf of the Atlanta district.

Investigation ongoing

The Pennsylvania study does not appear to have influenced the formal investigation into allegations of cheating, which was essentially rejected by the state. In August, Gov. Sonny Perdue deemed the investigation, overseen by an outside community panel, inadequate. He named two special investigators to look at the evidence. Their probe is ongoing.

Perdue has repeatedly rebuked Hall for being too slow to respond to the cheating allegations.

This fall, federal prosecutors opened a criminal investigation into whether district officials committed fraud by illicitly boosting scores on standardized tests.

The federal authorities are looking at whether Atlanta schools received federal grants based on inflated scores.

The AJC asked for the report in the weeks after the school system commissioned it in late 2009 and was told the analysis was not final. The report was not released until this week.

“The report was being used as another data point in their investigation,” Bromery said. “It would have been inappropriate to make the report public.”

The report cost $17,647, said officials from the Atlanta Education Fund, an outside agency affiliated with the system that paid for it with private funds. While the education fund paid for the study, Atlanta Public Schools requested it.

Khaatim S. El, an Atlanta school board member who recently became chairman, said Thursday he was not aware of the report and noted that Hall and two other board members are on the Atlanta Education Fund’s board of directors.

“I intend to ask ... why this information was never shared with the majority of the Board of Education or with the public,” he said.

In February, following the AJC’s articles, the state identified 58 Atlanta elementary and middle schools for possible cheating on state tests and ordered the school system to investigate.

1 in a billion odds

Questions about Atlanta’s testing program first arose in December 2008, when an AJC analysis found improbably steep summer retest gains at some schools. The state subsequently investigated six Atlanta educators from the city’s Deerwood Academy after it found evidence of an abnormal number of erasures on their students’ test answer sheets.

Last fall, a second AJC analysis showed 12 Atlanta schools posted highly unlikely gains or drops on the spring 2009 Criterion-Referenced Competency Tests, the state’s main academic measure for students in grades 1 through 8.

The schools included West Manor and Peyton Forest elementary schools, where students went from among the bottom performers statewide to among the best in one year. According to the AJC analysis, the odds against such a leap were worse than one in a billion.

Atlanta officials said at the time they did not believe cheating occurred. But, Hall announced she would seek outside help to pursue the issue.

Andrew Porter, dean of the University of Pennsylvania’s Graduate School of Education, analyzed test score data related to the 12 schools named in the AJC report. His team, according to report, received its first suitable data file Jan. 28 and additional data April 15, during the Atlanta district’s state-ordered internal investigation.

The data included student demographic information and allowed them to track students and their test scores from grade to grade. Because of federal student privacy rules, the AJC did not have access to this data.

Researchers looked at 12 factors, including enrollment, race, gender and economic status, to determine if student turnover or shifts in demographics could explain unusual test score gains or drops identified by the AJC analysis.

Of the 22 Atlanta classes and subject areas — such as a third-grade reading class — identified as unusual by the AJC analysis, the Pennsylvania researchers confirmed 14 test results as highly unusual even accounting for student turnover and demographics.

“In summary, controlling for shifts in student demographics from 2008 to 2009 ... did not in large part negate the newspaper findings,” the researchers wrote.

The turnover study also turned up unexpected test score gains in three classes not found by the AJC study.

“The possibility that enrollment shifts could be the reason behind the unusual test score gains and losses we found in our analysis was a concern we shared with school officials while working on our original story,” AJC data specialist John Perry said.

“So it’s good that someone was finally able to get access to the information needed to answer these questions.”

-------------------------------------------

How we got the story

Last fall, the AJC reported statistically unlikely gains in test scores at some Atlanta schools on the Criterion-Referenced Competency Test in 2008 and 2009.

● In addition, the Governor’s Office of Student Achievement commissioned an erasure analysis of 2009 CRCTs statewide, which found suspicious erasures on thousands of tests from hundreds of classrooms across the state, suggesting answers had been changed at a much higher rate than is expected. The results were released in February.

Both analyses looked at tests administered to first- through eighth-graders in three of the several subject tests: language arts, reading and math.

● Atlanta had 58 schools flagged, the highest number of any district.

The state Board of Education ordered 35 systems with suspicious erasures to investigate 191 schools statewide, to find out if and how cheating occurred. The results were due in May.

Atlanta named a blue ribbon commission to conduct its review, and that group hired consultants to do the analysis. The commission asked to extend its deadline to mid-June, then submitted its report to the state on Aug. 2.

● Questions were immediately raised about the approach and thoroughness of Atlanta’s report.

A majority of Atlanta Board of Education members initially voted not to accept the report, then approved it two weeks later.