With the Milestones scores coming in this week, I thought this issue was worth revisiting.
Along with thousands of other Georgia high school students, my twins are taking the Georgia Milestones this week. I never thought much about who grades these state tests until several parents and teachers reported seeing ads in local publications for graders placed by Data Recognition Corporation, Georgia's testing contractor.
The ad states:
Data Recognition Corporation is a national leader in educational testing and we are seeking temporary Test Scorers to assist us in our busy assessment season. You will be scoring standardized tests that children take to comply with the No Child Left Behind Act. These tests are administered to elementary through high school grades in the subjects of reading, writing and math. You must have a four-year degree to qualify for this position. The pay is $12.00 per hour plus eligibility to earn an attendance bonus, which would bring your rate to $13.25 hour. We offer paid training and a pleasant work atmosphere.
We wondered about hiring temporary graders so my AJC colleague Ty Tagami consulted testing expert Greg Cizek at the University of North Carolina at Chapel Hill. Cizek was an expert witness in the Atlanta Public Schools cheating trial.
Tagami says:
Dr. Cizek told me the testing company that placed ad is well-known and well-reputed. He also told me it would be standard practice to employ part-time scorers with undergraduate degrees and training by the company – so long as their degrees are in a subject related to the tested material.
A Fulton mother went well beyond just reading the ad; she applied for one of the jobs to score the student responses that cannot be graded electronically, the short answers and essays. Alarmed by her experience, the parent shared a detailed account with the state Department of Education and legislators. And now all of you.
Here is her account, slightly edited for space:
They asked me to bring proof of my degree by bringing a copy of the original diploma, official transcripts or an unexpired teaching license. At the job fair, they said the pay was $13/hour with a potential of a $50 bonus, so it would equal $14.25/hour. (Note from Maureen: Her bachelor's degree is in fashion merchandising.)
There were about 15 of us and they brought us to a room, handed us the application, a form stating the information we submitted was accurate, a math and English test that we had to take and two other optional forms to fill out. After sitting down, I got my phone out and took pictures of the math and English test. There were two girls who handed everything out and they were not paying attention so they never said anything.
I started to complete the four math questions and, honestly, did not know how to answer one. It asked if you had dough to fill a 10 inch x 14 inch pan, what size circular pan would you need to fill the same area and thickness. It's been awhile since I have done equations so I was unsure. I got my phone out and Googled the answer. No one said anything to me. They did a five-minute presentation about DRC and shared what the position would entail.
My takeaway from the presentation was that they are looking for people during the day from 9:30 to 4 with one unpaid 30-minute break and two 15-minute paid breaks. They were also looking for people in the evening from 5:30 to 9:30 and you would receive one 15-minute paid break. They said we would all be in a room together and sitting closely. They also said you could receive a potential $50 bonus if you showed up for your shift every day and that they would assess you on the quantity you graded.
From my understanding, you have to score/grade a certain number of tests within the time-frame that you work. In the presentation, it said you will be assessed on quantity. I then had a one-on-one interview in a break room where people were able to come in and out to get drinks and snacks. In that five-minute interview, I was asked about my previous employment and why I thought I was qualified for the job.
I was also asked if I had ever received criticism from a supervisor and how I handled it. I was then asked if I would be comfortable being in "close quarters" with the other test scorers. They also asked if I would be able to grade the same question over and over and over and over again.
I was told I would receive a letter in the mail letting me know if they would offer me the position. Please note this was on Thursday and the training begins next Wednesday. So, in six days, I assume they will look over my resume, check my references, verify I have a college degree and grade my math and English test and be able to get a letter in the mail in enough time to start training next Wednesday.
When they asked everyone for a copy of their college diplomas (or an official transcript or an unexpired teaching license), only about 6 of the 15 people had brought it. The employees at DRC told them not to worry that they would just need to see a copy before the training starts.
With all that being said, I am very concerned about the quality of the potential test scorers. For the Georgia Milestones to be such a high-stakes test and for all the work and time the teachers and administration have put into getting the children prepared and this is how they are going to interview potential test scorers.
I have never contacted anyone at the Department of Education but I feel so strongly about this that I could not just sit back.
The parent would have been grading tests from Ohio and Pennsylvania. The Georgia tests are graded in Indiana, Ohio and Wisconsin. Still, the parent said, "It concerns me the same thing is happening at another of their scoring sites where it will be graded.”
After the mom shared her experiences with Melissa Fincher, DOE deputy superintendent for assessment and accountability, Fincher responded with an email saying she planned to talk to DRC, adding, “...it concerns me as well which is why I have escalated your report to the president."
In another email, Fincher said:
I will share your email with the President of DRC directly as well as those responsible for their hand-scoring operation. The experience you describe is not acceptable.
Please note that our staff monitors, closely, the scoring process of our state assessment – given the stakes you mention. We travel to our scoring sites to observe scoring and monitor the scoring metrics (which are based on accuracy, not quantity!) each and every day. In addition, we are able to read behind scorers to ensure accuracy.
In response to questions from legislators this week about the qualifications of the temps hired to score the Milestones, DOE is sending out this statement; it differs a bit from the experiences of the Fulton parent.
Once scoring is underway, there are many checks that are conducted to ensure scorers maintain fidelity to the scoring rubric. Metrics are closely monitored at both the item and individual scorer level. We (GaDOE) receive daily reports. We also visit the scoring sites to monitor scoring. Georgia follows industry standards – and this type of scoring has gone on for years in Georgia with our former writing assessments. This process is not new.
Many of the scorers are retired professionals – many of whom are retired or former educators. This isn't the case of just anyone can score….there is a very detailed process that is followed from start to finish.
In talking to parents, many assumed former teachers graded all tests. Do the graders have to be educators? Does it matter to you?
About the Author
