“No one beta-tested” healthcare.gov.
Sam Stein during an interview Monday on MSNBC’s “Morning Joe”
The rollout of the Obama administration’s health insurance marketplaces website has been so roiled with problems — from a pricing glitch to flawed data to widespread failure of log-ins — that even journalists from left-leaning media are calling for top officials to be hauled before Congress.
But while appearing on MSNBC’s “Morning Joe,” Huffington Post political editor Sam Stein went a bit too far in criticizing the agency’s prep work before the Oct. 1 launch of healthcare.gov.
Or at least that’s what he told us when we asked him.
“It seems from all of the reporting, and from what I can gather, that up until very recently they (the Obama administration) didn’t realize how bad this was going to go out,” Stein said on MSNBC. “And no one beta-tested the site, which is almost criminal when you think about it.”
We wanted to check that out.
Stein directed us to a Washington Examiner report with a headline that somewhat contradicted his statement, which he acknowledged: “Troubled Obamacare website wasn’t tested until a week before launch.”
“Unfortunately, in the rush of cable news, I didn’t add the clause ‘until a week before launch,’ ” Stein told us. “I should have done that and apologize.”
That’s true. But Stein is more right than even he thinks, in part because he inadvertently used the word “beta” to describe the specific website testing. In the world of information technology, there are many kinds of tests under many kinds of names.
Put another way, Stein goofed — but by goofing, he got closer to the truth.
Here’s what you need to know:
The Examiner story, quoting someone speaking anonymously, said officials did not allow testing on the website until just days before it went live Oct. 1. The biggest problem, according to this report and others, was that leaders of the Centers for Medicare and Medicaid Services decided to act as the central coordinator for the project (CMS would not tell the Examiner whether the agency holds that role).
The next day, The Washington Post described how government officials and contractors proceeded with the rollout despite a botched crucial test days earlier. They ran a simulation, unsuccessfully, in which a few hundred people tried to log onto the website at the same time. Officials went forward with the launch, and the website “locked up” almost immediately when 2,000 users tried step one, the Post reported.
Unnamed sources told the Post that an end-to-end trial run of the process did not happen by as late as Sept. 26, the week before the launch.
“Beta testing” is a very specific term in the tech world that most professional software goes through before launching. It comes after a product is in an “alpha” phase, or the earliest version of software that is subject to some tests to find any big issues. When a product reaches beta, it is tested by a larger group of people not connected to its development in an effort to gather feedback and make more fixes before a large-scale launch.
Big tech companies such as Google sometimes launch products in beta mode, but enterprise and government software vendors don’t usually do it that way, said Alexander Howard, a former Washington correspondent for tech-centric O’Reilly Media and a fellow at the Tow Center for Digital Journalism at Columbia University. They stop adding features in the months preceding a launch to “start testing the heck out of it,” Howard said.
The development of healthcare.gov has been an entirely different story, he said.
“The only thing we have is reporting that says the first testing occurred the last week before it went live, and then it went live,” Howard said. “There was testing, but it’s not clear that it was in a beta version.”
According to published reports, healthcare.gov got additional features quite close to the launch, with no evidence of a consumer-centric test that went outside of the government. So Stein might have inartfully used the word beta-testing, but he might also have a point.
“Even if those tests nominally did occur, they were as good as nonexistent based on the complexity of the project,” said Patrick Ruffini, a Republican digital strategist.
Testing of any kind was difficult because the specifications for the website were evolving the month before the website went up, said Gail Wilensky, a senior fellow at Project Hope and former Medicare director under President George H. W. Bush.
The Post story, by the way, said a test group of 10 insurers that had access to the site advised the Centers for Medicare and Medicaid Services that the site was not ready to launch nationwide.
We reached out to CMS and CGI Federal, one of the main contractors for the website, but did not hear back.
Our ruling
Here’s what Stein said: “No one beta-tested” healthcare.gov.
Stein, in fact, meant to say something else, he told us. But at PolitiFact we check the statements as they are made.
In this case, Stein is closer to the truth than maybe even he thought.
Beta testing traditionally means certain members of the public were allowed to access the website well before it opened. There’s no evidence to suggest that happened.
Ultimately, this is a glitch of a statement talking about the glitches of a website.
But it rates Mostly True.
About the Author
