How test scores were analyzed

After months of stories about suspicious test scores in The Atlanta Journal-Constitution, the Governor’s Office of Student Achievement launched a wider review of 2009 results. In October, the AJC reported a dozen Atlanta schools had gains or drops on standardized tests so extraordinary they were less likely than picking the right lottery numbers.

The governor’s office contacted testing contractor CTB/McGraw-Hill to examine erasures on the Criterion-Referenced Competency Test, the statewide measure of student achievement in grades one through eight. The company agreed to do the analysis at no cost. CTB/McGraw-Hill reviewed student answer sheets from the spring 2009 CRCTs in three subjects — English, math and reading — for 1,857 schools statewide. An erasure analysis scans and tallies answers erased and changed from wrong to right.

The analysis covered 152,986 classrooms across the state. It counted the numbers of wrong-to-right changes per student, averaged by classroom, and compared the numbers with the state norm for such changes.

Made public Wednesday, the analysis flagged schools whose erasures were deemed unusually high, which was defined as three “standard deviations” above the statewide average. Experts consider that a conservative standard.

Standard deviation is a measure based on the distribution of results below and above the average. With a normal distribution, 99.85 percent of results are expected to fall less than three standard deviations above or below the average.

Schools were ranked in four categories, depending on the number of classrooms showing high numbers of changes.

Schools with 25 percent or more of classrooms flagged were placed in the “severe concern” category. Schools with 11 percent to 24 percent of classrooms flagged went into the “moderate concern” category. Those with 6 percent to 10 percent of classrooms flagged were placed in the “minimal concern” category.