As students return to school this fall, Gov. Sonny Perdue and the Governor’s Office of Student Achievement, or GOSA, continue to criticize the work of my company, Caveon Test Security. The governor’s position is misguided with respect to Caveon, but he is calling attention to an ugly reality: Wholesale, organized cheating in some Atlanta public schools occurred and must be addressed. His statements about Caveon, though, do not rest on well-established and scientific cheating detection methods.
The governor’s public statements about our analyses of 2009 statewide Criterion Referenced Competency Tests, or CRCT, data ignore that we confirmed GOSA’s conclusion that cheating occurred at some APS schools. Caveon is the leading test security company in the field, with experience examining more than 15 million test instances for the largest, most esteemed test organizations in the world, including 15 state departments of education and the Department of Defense.
The state’s vendor used a “preliminary screening” method of counting wrong-to-right CRCT erasures. Caveon would never recommend that our clients launch full-scale investigations solely on the basis of wrong-to-right erasures. Such an approach flies in the face of industry best practices.
A key concept in our work is helping clients focus on the “worst of the worst.” We also believe that minimizing false accusations is highly desirable. This means every school and class that is investigated shows clear evidence of serious problems.
What the governor seems unwilling to recognize is that many of the Atlanta schools flagged in the original GOSA analysis are almost certainly on the list of “schools of concern” because of factors unrelated to cheating. These factors include common missteps such as students getting off track and making mistakes marking their answer sheets. “Losing your place” on answer sheets occurs on a fairly common basis. The CRCT is no exception, and when a student realizes a miscue, erases the misaligned marks and corrects the answer sheet, wrong-to-right erasures are created.
No credible measurement expert would be satisfied with the original GOSA analyses as a basis for deciding that a teacher or school had serious problems unless the results were very extreme, as was the case with some of the schools flagged by GOSA and confirmed by Caveon’s in-depth follow-up.
To the detriment of our ability to analyze tests to the fullest extent possible, the GOSA provided data only for the 58 schools that had been designated as being “of concern,” not all APS schools. Most importantly, we did not receive student-by-student response data for every question. This omission prevented us from conducting these other cheating analyses:
● Collusion: Which schools and classes had “unusual agreement” among students, far beyond what could reasonably have occurred by chance alone? This is the most basic cheating analysis, one any expert would insist on doing.
● Unusual patterns: Which schools and classes had very odd patterns of student responses, such as missing easy questions and answering hard questions? This outcome is very likely to have been caused by students receiving help before or during an exam as well as tampering with answer sheets or student records after testing.
From Caveon’s perspective, we would very much appreciate discontinuation of uninformed and wrongheaded criticism of the state-of-the-art cheating analyses that we did. Any contention that we jeopardized our personal and professional reputations to please a client has no basis in fact whatsoever.
John Fremer is president and COO of Caveon Testing Security.
About the Author
