Get Schooled

By jim d

March 23, 2006 11:48 AM | Link to this

Patti,

Along with the increased demand for accountability for test takers there appears to be a growing need for the same accountability to be applied to those writing and grading the tests. Such an incident was reported by CNN just a week ago in regards to faulty test scoring on the PRAXIS and an 11.1 million dollar settlement with teachers that lost out due to faulty scoring.

http://www.cnn.com/2006/EDUCATION/03/16/testing.error.lawsuits.ap/index.html?section=cnn_education

I honestly expect to see a financial settlement reached with these SAT takers as well.

Indeed I believe that if high stakes testing is to ever legitimized, the scoring and production of such tests must be held accountable.

By Susie

March 23, 2006 11:54 AM | Link to this

It still blows my mind that something like passing a grade, graduating or getting into a college hinges on a single test score.

By high school teacher

March 23, 2006 12:00 PM | Link to this

And yet, the welfare of our state education is based upon our average SAT score. Wonder how may times this has happened before and no one found out?

The College Board has ceased to be about education. It has bowed to the almighty dollar. We need to encourage students to take the ACT instead; its focus is more similar to curriculum than the SAT.

By Atlanta Teacher

March 23, 2006 12:08 PM | Link to this

high school teacher: i agree. looking back, i wish i had taken the ACT instead of the SAT…i think i would have done better.

is ETS a non-profit? i’ve always wondered how much money they make, considering they have a monopoly on standardized testing.

By Nel

March 23, 2006 12:11 PM | Link to this

The problem is with the fact that this recently came to light. The same problem could have existed within ACT but because it’s not as “high stakes” as SAT we may not have heard about it. This should prompt a closer look at both.

Why do we have two different tests for college anyway? Or are we just keeping somebody’s company in business in perpetuity?

By jim d

March 23, 2006 12:16 PM | Link to this

AT,

Yes, according to the article linked above, ETS is non profit. Which would mean they can’t show a profit. But they can over pay executives and provide them with perks. I suspect there’s more than a few execs. at ETS earning a handsome salary.

By Robert

March 23, 2006 12:29 PM | Link to this

People are such hyprocites….

You make a mistake and you want everyone to forgive you.

Others make a mistake and you want to sue. People that work to create questions on any test can make errors. It is simply because people are involved and human error will exist.

As long as those errors are corrected, why the big deal?

Yes, the SAT is high stakes. But if the “SAT gods” correct the mistake and there is no harm, then why the issue at all?

If a student is denied admission into a specific college due to low SAT score and then it was that error that caused that student to not be admitted, then fine. But unless this series of events happen, it is just yellow journalism intended to stir up BS.

By Patti Ghezzi

March 23, 2006 12:32 PM | Link to this

Clarification: The College Board, which oversees the SAT as well as the AP program, is nonprofit, meaning it should reinvest profits in the community. ETS is a separate entity, also calling itself a nonprofit, that develops and adminsters the SAT and the AP exams. The SAT is an extra-large pie, though and it seems another test company, this one for-profit Pearson, has the contract to score the SAT. It appears Pearson made the mistake. ETS does not have a monopoly on the testing industry. Other test companies include Pearson and Riverside. The number of companies bidding for testing contracts is small, however, and many concerns have been raised over the years that the industry is ill-equipped to handle all the testing required under NCLB.

Patti

By jim d

March 23, 2006 12:33 PM | Link to this

It did happen Robert. Haven’t you been watching the news?

By Becca

March 23, 2006 12:34 PM | Link to this

One of my worries too is that so many states brag about what their students score on the SAT. Some states are scorned because of their students scores. Now how do we know which states are really doing good and which are not?

By jim d

March 23, 2006 12:43 PM | Link to this

Right you are Patti,

But since ETS is not involved in the scoring process it would be perfectly acceptable for execs from ETS to have personally invested in the companies that do provide that service.

Think it could have happened?

By Robert

March 23, 2006 12:43 PM | Link to this

From reading the article, the SAT errors are being corrected, and the college admissions officers have been contacted such that no student with an incorrect score will be affected.

Again I ask, if no party is “injured” then how come this is a big deal? Yes, a mistake was made, but it is corrected. A student will not be denied college admissions due to this mistake.

The mistake was in October. Students that took the SAT in October wouldn’t be admitted until fall of this year anyway. Everything will be corrected way before then. No harm, no foul (except some elevated stress levels).

By Nel

March 23, 2006 12:53 PM | Link to this

Robert, when this first broke a couple of weeks ago, and admissions person was interviewed and her comment was the they would review the applications and “see what they could do to accomodate affected students”. I don’t think that comment constitutes “they can still come here if they qualify”. So it’s not “no harm no foul”. That’s easy to say when it’s not your child who is affected. Remember the results are published within a couple of weeks of the tests being given.

By Patti Ghezzi

March 23, 2006 01:13 PM | Link to this

Jim, I’m not following your question. Did execs from ETS invest in Pearson? I don’t see why it matters, unless they are trading on inside information.

Pearson is a giant publishing company and test-scoring is a small piece of its operation. The stock is currently trading at $14 a share, a 52-week high it looks like. P/E ratio is around 7. Some people might see that as a worthwhile investment, though it’s hardly a Wall Street darling.

Test scoring isn’t a lucrative industry. It’s labor intensive, seasonal and requires expensive technology.

By high school teacher

March 23, 2006 01:17 PM | Link to this

Robert,

A student could have been denied admission to a college because of an erroneous SAT score. In some cases, it won’t matter if the university is contacted about the error because they have already granted admission to another student in the place of the one who was denied. Universities have limits on the number of incoming freshmen.

Also, many high schools award students for performance on their SAT scores. Most high schools have already names their STAR students for this school year. Do you rescind the title from an undeserving student, or do you tell the student whose scoresheet was “lost” too bad, so sad?

By jim d

March 23, 2006 01:20 PM | Link to this

I’m not sure I’m following my question myself other than wondering why the test errors came out and why only after it was too late for some students. Was money the motive for not letting it out until they did and with a test thats held as a secret why and how it ever came out in the first place? Who would have known the difference?

By Nikole

March 23, 2006 01:23 PM | Link to this

Robert- Being admitted to a school occurs in the fall and spring, and late admissions are in the summer. And the “harm” is that a test should not have such an effect on your college admissions and the like. If the mistake had happened to me, I might have still be accepted to my school of choice, but would not have been offered an academic scholarship, because your SAT score had to be a certain number and above. See nomoretest.com for other examples of gross misues and abuse caused by standardized testing.

By jim d

March 23, 2006 02:02 PM | Link to this

Unless I’m mistaken Pearson may use a third party company for test scoring. While Pearson does seem to have some quality control in place for both hiring teacher scorers and third party companies, I’m not too sure they really audit these third party companies for compliance unless a problem arises.

Has anyone got any information on that?

By Amazed (Independent Woman)

March 23, 2006 02:36 PM | Link to this

The bad thing about incorrect scoring with the SAT is you don’t get your test back. So, you can’t argue about an incorrect problem. I’m sure this has occurred before, but was kept on the down low.

By Patti Ghezzi

March 23, 2006 05:05 PM | Link to this

I visited Pearson in Iowa a few years ago. The company has additional scoring facilities in Duluth and somewhere in Texas, Austin maybe. They have invested heavily in the technology and in recruiting the part-time resources to score the tests at these facilities. Whether they outsource some of the work to smaller outfits, I do not know. They would still be responsible for the results.

They have Georgia’s contract for the End of Course Tests. The turnaround time is five days.

By jim d

March 24, 2006 08:05 AM | Link to this

Great article this morning in the Arizona Republic about this subject and how its being dealt with.

http://www.azcentral.com/arizonarepublic/local/articles/0324testscan0324.html

By jim d

March 24, 2006 08:10 AM | Link to this

This thing just keeps growing larger. 27,000 more tests not scanned for errors.

http://www.washingtonpost.com/wp-dyn/content/article/2006/03/23/AR2006032301655.html

By jim d

March 24, 2006 08:33 AM | Link to this

Patti,

It would appear on the surface that Pearson may not have “invested heavily” enough. And College Boards, comments that “nothing like this has ever happened before,” and “We are 106 years old and have a long history of excellence,” Is like a doctor saying he’s never killed a patient before. Maybe, maybe not. All it means is he’s not been caught before.

By Patti Ghezzi

March 24, 2006 09:19 AM | Link to this

Hey all, I’m not defending Pearson or the College Board. I’m just sharing what I know about how Pearson’s operation is set up.

By Becca

March 24, 2006 09:23 AM | Link to this

Thanks for the links, Jim D.

By SET

March 24, 2006 09:47 AM | Link to this

Robert is basically right.

If you wait till you are about to buy a house to check your credit scores you may lose the house while you fix the typical errors on your credit report. That’s why you check these things periodically or in advance of a major action.

With the SAT and LSAT scores you don’t wait till the last opportunity to take the test because you will have no chances to recover from this type of problem or the other types of problems common - like an emergency or a flat tire the morning of the test,

Another example are Police Academy students I once talked to who were 20 to 22 years old. They gave up their normal snow/water skiing the year they were going through the (1 yr evening) Jr College Police Academy. If they had gone as usual and there was any accident - even a sprained ankle - they would have been put out of the academy and lost the year’s work on starting their $60k/yr career. Most of them don’t think this way. The better ones (from the better families) do.

And we haven’t even talked about the difference between those who sign up for and pay for a review course for the SAT/LSAT tests and those who don’t.

So this morning I’m just not full of pity or sympathy for people who can’t make it on any of these tests, high school exit, SAT or whatever. People who throw away their options by waiting till the last chance or by not going to a course where they take enough practise tests to know what they should be scoring ane the ones vulnerable to the rare mishap. That makes them losers, unreliable, and prone to “having problems”.

The kind of people I don’t hire.

When I write about the public school kids being turned out being unable to take care of themselves and turning up in trouble too much I also include this subtle weakness. They are accidents waiting to happen. And they will always have a facially good excuse when their checks bounce, they show up late, or they are caught without something they needed to have. It always someone else’s fault too. And they have a litigation history by 22.

The Boys tend to have physical issues, accidents (recklessness) and such. The Girls have relationship issues - radically poor judgement in trust. I believe this is also our public school training. You don’t see the same trends in the local private school kids - if s**t is going to happen they are taught to make it happen to someone else.

By jim d

March 24, 2006 10:16 AM | Link to this

Gee SET,

Have a rough night? You appear to be in rare form this morning.

Lets just face a few facts here, shall we?

The public school system in the United States is a huge institution. It involves about $500,000,000,000 (five hundred billion, or a half-trillion, dollars), 50,000,000 students, 6,000,000 employees, 100,000 school buildings, in nearly 15,000 school districts.

No one can be familiar with, or comprehend, more than a minute part of the total, and almost anything that can be said about it is true somewhere.

While some refuse to admit it, there are some very good things occurring. And while the establishment defenders refuse to admit it, some atrocious things exist.

But the bottom line remains That while some students have a great educational experience it is no consolation to those who don’t. So rather than “making it happen to others” why don’t we roll up or sleeves and make it better?

Oh Yeah, Nearly forgot the infamous words of Woodrow Wilson: “If you want to make enemies, try to change something.” Good thing Blogs didn’t exist back then our the quote may have been; “If you want to make enemies, jump on a blog and voice an opinion.” :-)

HAGD.

By Nikole

March 24, 2006 10:54 AM | Link to this

SET- I took the SAT my junior and senior year, and did better the 1st time, but it could have easily gone the other way. I also had to save my own money in order to take these tests. My mother finally had the money to buy some SAT software for me in between the two tests. And most of my friends, who were in the same financial situation as myself, took the test once, their senior year. Your business would be at a loss if you chose not to hire any of them due to something as foolish as this. You have given us another reason why we should do away with high stakes testing.

By Robert

March 24, 2006 12:15 PM | Link to this

Guys, the “SAT gods” could find 100% errors at this time (in March) and if corrected the colleges will still make the proper corrections for admissions in the fall. By this time of the year, few slots are filled ‘in stone’ one way or the other.

If they correct the errors and send out the proper SAT scores, what is the harm? I just don’t understand the hoop-la of it all.

The only ones that might be affected would be the students trying to gain “early admission” and this is only a very few students applying to few colleges that even offer early admissions (for example, GA Tech does not offer this option). And, if those students don’t gain early admissions, and their SAT scores are corrected and are worthy, then certainly they will gain regular admission. So again - where is the harm?

People that are up in arms, if they are honest, are not upset about this particular incident. But rather, they just don’t like the concept of any test like the SAT or ACT or any other, and so they are taking this opportunity to bad mouth it. However, rational folks see through this BS.

By Robert

March 24, 2006 12:50 PM | Link to this

Nikole -

Okay…. pretend that you are a college. How would YOU determine who gets the limited slots available?

GPA? No, that would not really work. Some high schools are “easier” than others and so those kids would have inflated gpa’s compared to “harder” high schools.

Extra-curricular activites? No, some high school students may participate in everything and yet be poor academic students. This just won’t work by itself.

Teacher recommendations? No, students can forge these. Also, a middle-of-the-road student might be able to find one or two teachers willing to give a glowing recommendation. This won’t work either.

There just has to be some single and common measure of a students knowledge and ability in order for colleges to make a cognizant decision on which applicant gets the available slots. If anyone has any other bright ideas, I am sure that colleges would love to hear from you.

By David200

March 24, 2006 02:24 PM | Link to this

Guys, Guys, Guys,

Reality check time: How many of you have heard of the SEM (standard error of measurement)? How many have heard of Significant Figures? An individual student’s score can be off, plus or minus, 30 points from a previous score and still be considered the same score!

Statistics should be left to those who understand them and not bandied about by college admissions officers, politicians and journalists (sorry Patti but it’s true). There may be honest disagreement between statisticians (mathematicians) and scientists about the validity of school/state averages; however, there is consensus that a school/state increase (or decrease) of 3 points can be considered inconsequential. Scientists who use Significant Figures consider a difference of 10 points on the SAT average in any sample size inconsequential.

We are considering a tool (the SAT) that is valid only under vague conditions (College Aptitude) to be valid for a single specific topic (ability to do college work). Guys, everybody in the profession knows this, but they’re not going to tell you. Sheesh, no college admissions officer wants to really say, “Sorry dude. My “gut” tells me you won’t be able to do college work. No, I have nothing to back that up but my experience but that’s my judgment. That’s what they pay me for.” You’d sue his pants off. The SAT is a convenient way for him to get out of being sued.

By Nikole

March 24, 2006 03:45 PM | Link to this

Colleges already use more than just to a test score as criteria for admittance and they should continue to do so. That is not my issue. My personal issue was that some schools grant scholarships based on scores and that could be adversely affected. Fair Tests website has a list of school that do not require an SAT score though, and maybe their websites can tell you how they decide who is admitted.

By SET

March 27, 2006 10:09 AM | Link to this

The high stakes school testing debate has some similarity to credit scoring. The principles are the same.

The test scores (are believed to) accurately predit future success and failure. The losers don’t like it when their loser tendencies are exposed. On a case by case basis their may be excuses and accidents on one person’s scores.

If you have a scoring problem you should have detected the issues and fixed them before you ran out of time. Oh! That means someone has to take personal responsibility for what happens to them… well that eliminates some people!

The lenders and the colleges are under time pressure too and don’t have all day to make exceptions for an individual. That’s why the scoring systems have been deveoped. It makes it possible to sort and decide on large groups of people quickly and cheaply. The system is not perfect but look how many people are resolved fairly compared to the number of errors.

And this is a trend that is continuing. Actuarial scoring is poping up all over our lives. Revenge of the math majors!

Brave new world!.

By SET

March 28, 2006 10:31 AM | Link to this

Nikole, Robert, et al.

My memory was just jugged by a poster here.

My law school’s head of admission was a statistician - She was there for some years and moved on. I had the occasion to work with several applicants on getting them in and met with the Dean and the Admissions people in the past. They explained how they took (10,000-20,000?) applications from across the country and selected a new class in the months between the arrival of the application and the mailing of the admissions decision.

It was a very daunting job with a lot of deadlines. They have shifted the incoming to electronic form to some extent - otherwise as soon as an application arrives it is keyed into the computer. The software they have then tells them what they have in the pool to work with. Then the winnowing out process begins at a fast pace. The computer is told to be on the lookout for certain rarities and those are flagged. People who are older, younger, or come from the one state with no alumni (“next app from that state gets in”) - people of unusual distinction (we once had a Playboy pinup I think) people with rare undergraduate majors, Military academy graduates, the list went on and on. There were relatively few applicants available on the special watch list.

Since they don’t want the school to fill up with applicants from the same zip code the computer applies a geographic spread to the pool which means that Los Angeles will likely get more turn downs than Redding CA for 4.0 students. The used to have Black seats and Women’s seats that were protected - applicants of thoses types only competed with each other for those seats. Affirmative action is illegal now in CA Schools and there are no more protected racial seats. There are still protected seats for applicants from certain states I believe (school financing was involved in this, I believe).

The final class arrangement to some extent is predetermined. The helps computer finds those people among the pool. When the admission committee is working a group of applicants they are trying to fill a certain block of seats and quickly winnow a block of applicants down to the openings at hand. As the first round of acceptances go out refusals come in (Because some better or more convenient school got to them first) and the rejected admissions lead to letters for the next layer of prospects.

As you see this process does not lend itself to a lot of personal interviews, or anything time consuming. The students they really want are the same the other schools want - the top scoring brilliant applicants perhaps already connected politically or professionally.

The LSAT was the first attempt. At certain scores it’s very accurate in predicting who will or will not graduate and practice law.

The process has trouble (so far) stopping psychopatic, bi-polar, drug using, damaged goods from getting in as long as those persons have kept up with the pack otherwise. These personalities are either kicked out during the degree program or stopped before bar admission by the backgroung checks, A significant portion of the class will never be allowed to practice law because of personal problems. Law Schools attract dangerous personalities who can hide in the large pool - just like school staff applicants or police applicants. The schools are trying to develop indexes to identify these people and stop them from getting into the schools in the first place.

I expect secondary score systems to creep in which will better serve as a stop on enrollment. The State Bar is also able to analyze those who were disbarred and suspended to determine what personal traits distinguish the losers from normal practitioners - then develop a scoring system for the background process.

Those on the receiving end of these rapid scoring systems usually blame everyone but themselves for not being “given” an opportunity.

Too Bad, so sad. You have to deal with reality before it deals with you. You can’t do these admissions jobs without rapid scoring. There are too many applicants and too little time and staff.

My applicants usually got in because I learned everything I could about the process and we spent time, money and energy getting the applicants ready (making them more attractive).

And yes, the Reese Witherspoon character in Legally Blonde would have grabbed the attention of the admissions team. She was unique - otherwise qualified, and she had a lot of nerve and charm. The videotape was a killer!

Brave new world!

By Nikole

March 28, 2006 11:01 AM | Link to this

The whole process is very interesting. Especially the part about not accepting too many people from the same geographic area. Ironic how upset some get over affirmative action, yet have no problem giving up their seat to a similarly qualified neighbor, just because they live near each other. I think that’s something I’m going to have to read more about.

By SET

March 28, 2006 12:29 PM | Link to this

The zip code problem (which is what I call the Los Angeles problem) refers to the fact that with so many people - If you accept only based on GPA and test scores, LA can drown out the rest of the state.

Since the state school gets it budget from the legislature they feel that geographic balance has to be considered along with all the other things they want the entering class to be.

The protected seats from certain states I believe refers to old agreements with those states that they contribute to our school’s budget and we take a certain number of their candidates. If this arrangement is still going on.

My point is that all of these factors have to be taken care of in a short period of time with a massive number of applicants from across the country - and with a reasonably small admissions staff.

Thus the need to use computerization and profiling of the candidates (and scoring on any number of indexes) to rank the applicants.

The same process is getting into many things we go through now - insurance purchasing to mortgages and borrowing to job placement. Imagine GM trying to figure out who to fire today.

So it’s incumbent on people to find out what is happening around them and to be ready for it. Thus high school students who think they might want to go to college (and have a choice which one) should take the screening tests (maybe?) early and often - and use prep courses if they possibly can.

With Affirmative Action going out the window there is still plenty of ways for minority or non-traditional (older?) candidates to make it but they have to get their act together and plan. AA reduced and eliminated the need for this. For example there were only 1000 competitive black candidates for law school a year with 452 ABA accredited schools in the country.

The numbers for Native American and Chinese or Japanese American candidates annually are small also. School fought over them.

But with AA gone, the UC Berkeley School of Engineering no longer has protected seats for WASPs. Guess what has happened??

Brave New World!