The job conditions for President Barack Obama’s next defense secretary have already spurred some top contenders to bow out, leaving the White House with a slim list of candidates to fill the post for the administration’s final two years. On Monday, Homeland Security Secretary Jeh Johnson became the latest to tell the White House he wasn’t interested in the job, according to people familiar with the process.

Johnson’s decision to stay put at DHS is likely to deepen the impression that the Pentagon post — typically a highly sought-after Cabinet spot —is drawing little interest. The president’s short list of contenders now includes longtime public servants who have deep Pentagon experience, but may be less likely to give Obama’s national security agenda the jolt that critics — and increasingly some supporters — say is needed.

Top contenders include Ashton Carter, the former deputy defense secretary who left the administration in late 2013, and Robert Work, who now holds the Pentagon’s No. 2 job.

That’s a far shorter list than some in the White House had hoped when Defense Secretary Chuck Hagel resigned last week under pressure from Obama. Michele Flournoy, one of Obama’s top choices, quickly took her name out of contention, in part because of concerns over the tight rein the White House has tried to keep on the Defense Department. Sen. Jack Reed, a Rhode Island Democrat and West Point graduate, also made clear within hours of Hagel’s resignation that he wasn’t interested.

Defense analyst Anthony Cordesman said that as Obama approaches the end of his presidency, the Cabinet post is “not particularly desirable” for anyone with broader political ambitions.

“It’s very unlikely you will get political visibility or credit for being the secretary,” said Cordesman, who works at the Center for Strategic and International Studies. “There are just too many problems and uncertainties.”

Among them: questions about the effectiveness of Obama’s military campaign against the Islamic State, Russia’s continued provocations in Ukraine, tensions between the White House and Defense Department over closing the Guantanamo Bay detention center, and concerns at the Pentagon with the impact of deep spending cuts.

Hanging over all of those policy concerns is the uneasy relationship between the White House and the Pentagon throughout Obama’s six years in office. His first two defense secretaries, Robert Gates and Leon Panetta, have been bitingly critical about White House efforts to micromanage the Pentagon. And Hagel is said to have grown frustrated by the White House’s drawn-out policymaking process and lack of clarity in the president’s eventual decisions.

Lawrence Korb, a former assistant defense secretary, said that with just two years left in Obama’s presidency, the next Pentagon chief will be hamstrung in efforts to shape both the administration’s policies and the department’s relationship with the White House.

“The clock is ticking in terms of being able to make significant changes,” said Korb, a senior fellow at the Center for American Progress, a think-tank closely aligned with the White House.