As a small coterie of grim-faced advisers shuffled into the Oval Office on the evening of Oct. 15, President Barack Obama’s chief domestic accomplishment was falling apart 24 miles away, at a bustling high-tech data center in suburban Virginia.

HealthCare.gov, the $630 million online insurance marketplace, was a disaster after it went live on Oct. 1, with a roster of engineering repairs that would eventually swell to more than 600 items. The private contractors who built it were pointing fingers at one another. And inside the White House, after initially saying too much traffic was to blame, Obama’s closest confidants had few good answers.

The political dangers were clear to everyone in the room: Vice President Joe Biden; Kathleen Sebelius, the health secretary; Marilyn Tavenner, the Medicare chief; Denis McDonough, the chief of staff; Todd Park, the chief technology officer; and others. For 90 excruciating minutes, a furious and frustrated president peppered his team with questions as he struggled to understand the scope of a crisis that suddenly threatened his presidency.

“We created this problem we didn’t need to create,” Obama said, according to one adviser who, like several interviewed, insisted on anonymity to share details of the private session. “And it’s of our own doing, and it’s our most important initiative.”

Out of that tense Oval Office meeting grew a frantic effort aimed at rescuing not only the insurance portal and Obama’s credibility, but also the Democratic philosophy that an activist government can solve big, complex social problems.

The story of how the administration confronted one of the most perilous moments in Obama’s presidency — drawn from documents and from interviews with dozens of administration officials, lawmakers, insurance executives and tech experts working inside the HealthCare.gov “war room” — reveals an insular White House that did not initially appreciate the magnitude of its self-inflicted wounds, and sought help from trusted insiders as it scrambled to protect Obama’s image.

The urgent race to fix the website — now playing out behind the locked glass doors of the closely guarded war room in Columbia, Md. — has exposed a deeply dysfunctional relationship between the Department of Health and Human Services and its technology contractors, and tensions between the White House chief of staff and senior health department officials. It strained relations between the Obama administration and the insurance industry, helped revive a Republican Party battered after the two-week government shutdown and frustrated, even infuriated, congressional Democrats.

The rollout

The early reports were encouraging as HealthCare.gov opened for business on the morning of Tuesday, Oct. 1.

The long-planned federal web portal — envisioned as an online marketplace where consumers could shop for plans, compare coverage and determine whether they qualified for subsidies — was central to Obama’s promise of affordable care. (There are also 14 state-run exchanges.)

But in Herndon, Va., at the offices of CGI Federal, the American subsidiary of a Montreal-based information technology firm that built the bulk of the site, technicians were frantic. They were beginning to realize what the White House did not: that the exchange’s problems involved much more than delays caused by high traffic. Errors were popping up everywhere. Software that assigned identities to enrollees and ensured that they saw only their own personal data, known internally as the EIdM, was being quickly overwhelmed. Customers could not log in to create accounts.

Park was dispatched to help. A Harvard graduate and a son of Korean immigrants who co-founded a health information technology firm when he was 24, Park had the job of promoting innovation. Now, he and the software engineers who built the system were desperate to figure out what was wrong.

“They kept looking, looking, looking, but there wasn’t anybody moving through the system,” a person who worked on the project said.

Account creation was the province of Quality Software Services Inc., or QSSI, a company based in Columbia, Md. Its subcontractor, Oracle, flew a high-level team of software engineers to Washington. Experts disagree on what went wrong. But several said errors in the software code written to stitch the Oracle product into the online system and improperly configured hardware trapped users in endless technological loops. It would take eight days to resolve just that one bottleneck.

On Capitol Hill, lawmakers were consumed with another problem: the looming threat of a government default. The House Democratic Caucus gathered in the East Room of the White House on Oct. 9; Obama, participants said, vowed to hold the line with Republicans on the debt fight and assured nervous Democrats that his team would get the health portal working.

That same day, McDonough met in his office with Jeff Zients, a multimillionaire management consultant who had developed a reputation as a troubleshooter while running the Office of Management and Budget and is scheduled to become Obama’s top economic adviser in January.

A mad scramble

Chaos and frustration among the engineers was growing as fast in mid-October as the list of problems they were supposed to be fixing. Across the country, insurance executives were alarmed. Almost no one was buying their products.

In Herndon, as engineers tried to come to grips with repeated crashes, a host of problems were becoming apparent: inadequate capacity in its data center and sloppy computer code, partly the result of rushed work amid the rapidly changing specifications issued by the government.

The website had barely been tested before it went live, so a large number of software and hardware defects had not been uncovered. Fixing the account creation software simply exposed other problems.

After the big damage control meeting in the Oval Office, McDonough and Sebelius went to meet with the exhausted and disheartened staff at the Medicare agency. McDonough broached the idea of having an outsider take charge.

“Look,” he remembered telling Sebelius, “we’ve always recognized that as a management technique you’d always want independent eyes if we ran into a problem. What do you think about Jeff Zients?”

Sebelius hesitated. “Let’s think about it,” she said, by McDonough’s account.

It did not take much prodding; by the end of the ride, the secretary had agreed.

Zients decided the site needed a “systems integrator,” a single company that would take charge. On Oct. 24, Tavenner put Quality Software Services in that new role — a move that, people familiar with the project say, began to resolve conflicting and contradictory directions from her agency. But the bigger problem was organizational.

“People looked like they were busy,” said Andrew Slavitt, group executive vice president for QSSI and its parent company, Optum, “but it was hard to tell what they were working on and how it fit in.”

But while the contractors were grateful to Zients for helping to create order, they saw the administration’s “tech surge” — announced by Obama in the Rose Garden a few days before QSSI took over — as mostly an exercise in public relations.

The promise

Obama, meanwhile, was under assault. After years of telling Americans, “If you like your insurance plan, you can keep it,” he was being accused of lying.

Inside the West Wing, officials knew the political controversy had moved beyond the broken website. Now it was about a broken promise.

On Oct. 30, the president flew to Boston to talk about the Affordable Care Act at an event in Faneuil Hall, the Colonial-era meeting place where Mitt Romney, then the governor of Massachusetts, signed his own health care overhaul into law in 2006.

In addition to pledging again to fix the website, Obama for the first time acknowledged that not all people would be able to keep their health insurance. “For the vast majority of people who have health insurance that works, you can keep it,” he told the crowd. “So if you’re getting one of those letters,” he advised, “just shop around in the new marketplace.”

Aides hoped the admission would cool down the controversy.

But back in Washington, the president’s adversaries had other ideas. As senior Republican lawmakers huddled in strategy sessions to take advantage of the website debacle, their constituents began sending stories about having their health insurance canceled suddenly. Their anger at the president was palpable — and usable.

Bruno Gora, a 61-year-old self-employed promotional products distributor in Henrico, Va., for one, dashed off a note to his congressman, the House Republican leader, Rep. Eric Cantor. Cantor had for years been questioning Obama’s “If you like your plan, you can keep it” promise. Now there was tangible proof that the president had been wrong.

Anxious Democrats increased the pressure. Even former President Bill Clinton casually suggested in an interview on Nov. 12 that Obama should let people keep their insurance, even if it meant changing the law. And by the next Wednesday, with no change yet announced by Obama, Democratic lawmakers were in a full-blown panic.

Despite lingering concerns inside the administration about the long-term impact on the health care law, the president announced his solution: insurers would be allowed to renew old plans for a year.

The next steps

As the political debate raged on an hour away in Washington last week, the small group of technical experts that Zients assembled in Maryland focused on identifying and fixing the hundreds of software and hardware malfunctions that were bringing down the site and making it inaccessible.

In the White House, aides to Obama know that a clearer assessment of the Affordable Care Act will not come until at least the end of March, when the initial sign-up period for enrollment closes.

Obama, meanwhile, is trying to turn the page. At a closed-door fundraiser Tuesday night at the Beverly Hills home of basketball star Magic Johnson, Obama made only scant reference to the law that he has long hoped will define his presidency.

“I’m absolutely sure we’re going to make sure this country provides affordable health care for every single American,” Obama told the donors. “And if I have to fight for another three years to make sure that happens, I will do so.”

He did not mention the website.