But during a news conference in Australia Saturday, the president tried to put the comment in perspective.
Obama said, "The fact that an adviser who was never on our staff expressed an opinion that I completely disagree with in terms of the voters is not a reflection on the actual process that was run."
While it's true that Gruber was technically never part of Obama's staff, Politico pointed to White House records showing the consultant visited the White House a dozen times between 2009 and this year.
The comment has given major ammunition to pundits claiming the Obama administration intentionally misled the public on its health care law.
Then again, there's been a lot of that on both sides.
STEPHEN COLBERT, VIA COMEDY CENTRAL: "Whereas Republicans respected the voters' intelligence by telling them the death panels would grind up your grandma to make glue."
Gruber did show remorse on MSNBC after the video had taken off. "I was speaking off the cuff. I basically was speaking inappropriately, and I regret having made those comments."
But the damage is done, and it comes at a crucial time for health care reform. Earlier this month, the U.S. Supreme Court announced it would hear a new challenge to the Affordable Care Act.
The challenge questions whether subsidies should be available to all Americans who qualify for insurance established by the state.
The Washington Post called it, "most serious challenge to the Affordable Care Act since the justices found it constitutional more than two years ago."
The Affordable Care Act began open enrollment for 2015 on Saturday, taking in 23,000 completed applications for insurance in its first eight hours.
This video includes images from Getty Images.