Georgia’s test monitoring not as tough as other states

After allegations of widespread cheating on standardized tests made news in Texas nearly five years ago, state officials formed a battle plan.

They would study blocks of test questions to detect answer copying, assign monitors to campuses with a history of irregularities and punish cheating school districts by lowering their accreditation ratings. Officials promised to develop a statistical measure to scour hundreds of thousands of scores for the sort of impossibly steep gains that are a hallmark of blatant cheating.

Meanwhile, South Carolina’s education department for years has scrutinized erasures on answer sheets to look for unusual numbers of corrected responses. Suspicious findings are reported to state law enforcement. Other states also search yearly for patterns that would betray cheating.

But in Georgia, education officials have never performed such routine checks on state test scores, relying instead on whistle-blowers to signal wrongdoing.

A cheating scandal at four schools that has led to criminal charges has not prompted state officials so far to get tougher.

They say they will not systematically check standardized test results each year, although they will spot-check some unusual testing gains through a new audit process. The probe that identified the four schools was narrow, excluding more than two dozen other campuses with statistically unlikely increases, the AJC found.

Without examining scores more broadly, or taking a closer look at test security policies, it will be hard to determine how much cheating is taking place in Georgia —let alone stop it, one expert said.

“It’s just the tip of the iceberg, I think,” said Tom Haladyna, a professor emeritus at Arizona State University and testing expert. “The other 80 percent is being hidden.”

School leaders arrested

Last week, police arrested two administrators at Atherton Elementary, the school with the most unusual rise in scores. Principal James Berry and assistant principal Doretha Alexander face charges of falsifying a state document.

An AJC analysis last December disclosed that Atherton and a handful of other schools made statistically improbable gains on standardized retests last summer. Middle and elementary school students take Criterion Referenced Competency Tests each spring to measure basic academic skills. Students who fail can retake some tests during the summer.

In response, the Governor’s Office of Student Achievement examined erasure patterns on fifth-graders’ answer sheets at six schools from the 2008 retests. They found four with unusually high numbers of answers changed from wrong to right. School employees were also interviewed.

The achievement office called the evidence of tampering overwhelming and said it would refer school employees’ questionable conduct to the state agency that polices educators’ licenses.

Next month, the state board of education will consider invalidating the schools’ test scores and changing their status from passing to failing to meet federal standards, or make Adequate Yearly Progress. The schools only met the measure because of the jump in retest scores.

The criminal charges could be a first for Georgia, where the few educators caught tampering with tests have usually received punishments ranging from reprimands to suspensions.

But the scandal hasn’t inspired the state to take a significantly harder stance on cheating.

Executive Director Kathleen Mathers said the seven-person student achievement office will probably look at some test scores every year. But because there is so much data and so many ways to mine it, the office will probably choose one approach at a time – looking at one grade, for instance, or one type of standardized test. The office will also scrutinize other measures, such as the graduation rate.

“We operate from the perspective that student achievement data tells a story about students,” she said. “Anytime that story doesn’t make sense we’re going to look at it more closely.”

She said she didn’t know how many audits the office would do a year because it will not work under a quota. The office’s first audit took place last year, focusing on attendance rates at four Clayton County schools.

Unlike some states, Georgia won’t submit answer sheets regularly to erasure analysis or use statistical programs to routinely look for the sort of off-the-charts gains that could indicate cheating each year.

Rooting out cheaters

Twenty-three states have some sort of verification process, Mathers said.

South Carolina hires the contractor that scores its tests to perform routine erasure analyses as well, said Liz Jones, director of that state’s office of assessment. If irregularities are found, state monitors show up unannounced in teachers’ classrooms the next year.

The state reports suspicious findings to the school district involved and the South Carolina Law Enforcement Division for further investigation. South Carolina law makes it a misdemeanor to tamper with state tests, punishable by a fine of up to $1,000 and jail time, Jones said.

Louisiana does erasure analysis yearly, too. Some states have hired test security companies such as Utah-based Caveon to combat cheating.

The company has worked with 11 states, Vice President of Marketing Don Sorensen said, either tightening their security policies or running test results through Caveon’s forensic data analysis.

“There’s always some instance of cheating,” he said. “Sometimes it’s more widespread than others.”

In Maryland, the state combines some score trend studies with occasional erasure analysis, a process for reporting irregularities, and strict test-handling policies to police test tampering, said Leslie Wilson, a state assistant superintendent there. Some educators have lost their jobs because of what was found.

“We’ve always taken a hard line on it, which means people don’t take chances anymore,” she said.

Haladyna, the Arizona State expert, said probes looking for statistical patterns tend to be the most dependable for uncovering cheating. Even erasure analysis can be superficial, he said, because it won’t detect other forms of cheating – such as simply giving away answers.

More states study erasures than perform statistical analyses, said Gregory Cizek, a University of North Carolina testing expert. Many do neither.

Hiring companies like Caveon to crunch data can be expensive. “I don’t know that this is the time we’re going to be able to contract with them, but we’re certainly open to the conversation,” Mathers said. The state plans to meet with the company’s representatives in upcoming weeks.

The current probe looked only at fifth-grade gains on the retest. It considered schools that were above a statistical threshold – 3.5 standard deviations from the mean. Consequently, it singled out only extreme outliers: just six schools fit that category.

A common statistical measurement, standard deviation is used to see how much schools’ score increases differ from the average gain.

In her address to the state board, Mathers said research has shown that improving scores by two standard deviations without cheating is difficult. “A lot of researchers say just to get to two you have to have one-on-one instruction,” she said, calling the 3.5 standard “very generous.”

The AJC’s analysis found 30 schools with gains two standard deviations or above. Haladyna said even improvements more than one standard deviation deserve a second look. Changing fewer answers strategically can be done easily, he said, without triggering gains that send up a clear red flag.

Mathers said the state focused on the extremes in part because the investigative follow-up work would have taken a long time to finish if more schools were included. The number of students who took the retest was small, which might make it easier to obtain gains of two standard deviations legitimately if a few students did better than expected, she added.

She said the state may do broader investigations in the future.

“I think it sends a very loud message to our schools,” Mathers said. “That we can do this work and will do this work so that we make sure kids are getting services the right way.”

Some states don’t know what to do next when they find suspicious scores, Cizek said. Ideally, the problem is identified immediately when scores are tallied, the students affected take retests and an independent agency investigates, he said. Leaving it up to districts to examine cheating within their ranks is not adequate, he said.

When cheaters are caught publicly, he added, others take note.

“All it takes is a few high-profile cases for people to get serious about this,” Cizek said. “Most educators are doing this ethically and playing by the rules, but it is very important to catch those that aren’t.”

Data analyst John Perry contributed to this report.