Atlanta test-cheating probe fails to satisfy

By Alan Judd and Heather Vogell


The Atlanta Journal-Constitution

Investigators chosen by Atlanta’s school system gave only cursory attention to evidence suggesting extensive cheating on standardized tests in more than half the 58 schools they examined.

Seventeen schools suspected of some of the most widespread cheating were barely investigated and, consequently, avoided recommendations for sanctions. Another 14 schools where state officials voiced a moderate concern about cheating received similar treatment. The investigators disregarded testing irregularities in hundreds of Atlanta classrooms.

A commission that studied questionable gains on the 2009 Criterion-Referenced Competency Test, or CRCT, reported this week that alleged cheating seemed to permeate just 12 Atlanta schools, rather than the 58 — more than half the schools in the district — state officials flagged last winter.

The commission recommended possible disciplinary action against 109 Atlanta Public Schools employees.

Commission members defend their work. They say they focused mostly on schools highlighted in a statistical analysis performed by a consulting firm they hired.

But a review of the commission’s report and interviews with education officials and testing experts suggest that the investigation fell far short of unearthing the scope of a cheating scandal that calls into doubt a decade of higher test scores and other academic progress by Atlanta students.

“Our expectation was all 58 [schools] would be rigorously and thoroughly investigated,” said Kathleen Mathers, executive director of the Governor’s Office of Student Achievement, which is overseeing the state’s review. The investigation should have continued, Mathers said, “until you’ve cleaned up the problem.”

In particular, the Atlanta commission did little to investigate 17 of the 43 schools where state officials had found excessive erasures on test papers in 25 percent to 51 percent of classrooms.

At Deerwood Academy, for instance, the state flagged almost half of 90 classrooms. The Atlanta commission’s investigators noted unusual numbers of erasures and 100 percent pass rates on two tests given by one teacher. But the investigators interviewed just four people at Deerwood, cleared the entire staff and submitted a report that omitted the fact that an earlier investigation found strong evidence of cheating on a CRCT retest there in the summer of 2008.

At other schools, investigators spoke to as few as two staff members. In the case of one recently closed school, they spoke to none.

Even strong statistical suggestions of cheating and specific allegations did not prompt additional inquiries.

At Humphries Elementary, for example, where the state had flagged nearly half the classrooms for examination, a staff member told investigators that CRCT administrators “cleaned up” test papers in the school auditorium. Yet investigators spoke to only three people at Humphries and reported no educators for possible transgressions.

Scrutiny of Atlanta’s scores on the CRCT, a key indicator of whether schools meet federal and state standards, began after The Atlanta Journal-Constitution published analyses in 2008 and 2009 showing improbable increases in test scores at many schools in Atlanta and in other Georgia districts. In February, the state ordered Atlanta and 34 other districts to investigate schools found in its analysis to have excessive numbers of erasures on test papers.

The commission was appointed by the Atlanta Board of Education and populated by people with financial and civic ties to the district. By relying heavily on the work of its consultant, Caveon Test Security, the panel took an approach that seems to have conflicted with instructions from the state.

Instead of rigorously investigating all 58 schools, the commission gave a lower priority to 31 schools after Caveon reported that its own data analysis turned up fewer concerns than the state had reported.

Many of the 31, however, displayed the hallmarks of altered scores: classrooms or entire grades with suspicious erasures across multiple subject tests; classes with hundreds of erasures when only a few dozen might be expected; double-digit drops in 2010 test scores, under heavy scrutiny; and tips from other educators who suspected wrongdoing.

State officials are reviewing the report. So far, they aren’t happy with the results.

“I would not say I take everything Caveon says as the way it is,” Mathers said. “The state did an analysis, and we are confident in that analysis.”

The state found “significant variances that have to be explained” classroom by classroom, Mathers said. “We would expect that interviews would take place with all staff that had anything to do with the flagged classrooms.”

Atlanta school officials, as well as Superintendent Beverly Hall’s supporters, depicted the commission’s report as an exoneration of the district as a whole, an indictment of just a few schools and educators.

At a news conference Wednesday, the commission’s chairman, Gary Price, gave a more nuanced interpretation.

“We believe a thorough investigation was done on 58 schools, not just 12,” Price said. But Caveon’s analysis, he said, showed an “unbelievable” gap of questionable gains between the 12 schools and the remaining 46. So the commission restricted its toughest scrutiny to the 12.

“How long and how far do you look for every last person who potentially may have done something wrong?” Price said. “How many resources do you divert? How long do you keep the clouds over the schools’ heads?

“Or do you really go after the bad actors, work diligently to get that done, implement the recommendations for improvement that were suggested and move on? That’s the path the commission took.”

Price acknowledged that while the investigation made a strong case against the 12 schools, it did not clear others of suspicion.

In fact, the commission’s report contains numerous examples of irregularities at most of the 58 schools.

In February, the state flagged 39 percent of classrooms at Thomasville Heights Elementary for suspicion of cheating. Later, a teacher from King Middle School came forward to say students promoted from Thomasville Heights and six other schools could not possibly have attained their scores without help. She gave investigators diagnostic test scores supporting her claims on about 40 students.

The state found suspicious wrong-to-right erasures on numerous Thomasville Heights test papers.

One fifth-grade class with 15 students, for instance, racked up 139 such erasures; the state average for a class that size was 27. When fifth-graders took state tests under close monitoring this spring, the proportion of students who failed math increased by 31 percentage points.

Yet Atlanta’s investigators interviewed only two Thomasville Heights staff members. They recommended none for possible disciplinary action.

In elementary schools such as Dobbs and Fickett, the Atlanta investigators noted a high number of erasures with 100 percent pass rates in some classrooms. The state had flagged tests in all three CRCT subject areas — math, reading and language arts — in multiple grades at both schools. But investigators interviewed only three people at each school and recommended sanctions against no educators.

At Finch Elementary, investigators interviewed three people and referred two for disciplinary action — apparently, only after a staff member reported that she was told to lie by claiming test materials were stored in a vault instead of the conference room where they actually were kept.

Thomasville Heights, Dobbs, Fickett and Finch were among the 43 schools about which the state expressed “severe” concerns in February. In Caveon’s analysis for the Atlanta investigation, however, all were among the 31 schools where “potential testing irregularities were not as probable,” according to the investigative report.

Caveon examined not just wrong-to-right erasures, but the total number of erasures, the change in test scores and the difference the changes made in the number of students who passed or failed.

Caveon’s analysis may have missed signs of cheating, however.

The state gave Caveon access only to test data from the 58 flagged schools, so the firm could not compare typical patterns of erasures among all Atlanta students.

“Statistically, this data restriction would have the effect of reducing the number of schools flagged by reducing variability in the data set,” said Gregory Cizek, a testing expert and professor at the University of North Carolina.

Further, because Caveon looked for suspicious increases in scores between 2008 and 2009, Cizek said, its analysis may have inaccurately ruled out schools where cheating had taken place over a number of years and test scores had not skyrocketed just on the 2009 CRCT. “They might still have been doing something inappropriate,” Cizek said of those schools. “But if they had been doing it a long time, it might not rank high on the composite.”

Even at schools where Caveon minimized suspicions of cheating, another indicator — steep declines on the 2010 CRCTs — suggests something was awry in 2009.

Math scores fell in 53 of the 58 schools flagged by the state, Price said. At numerous schools the commission listed among the least likely to have cheated, nearly every grade posted lower scores in nearly every subject.

Still, Price said, the 2010 scores “corroborated” the commission’s work.