House Speaker Paul Ryan, R-Wisconsin, on Tuesday scolded Donald Trump — without directly naming him — over the billionaire's refusal to sufficiently disavow the support of former Ku Klux Klan grand wizard David Duke.

Even so, moments later, Ryan insisted to reporters at the Capitol that he still planned to support the Republican presidential nominee no matter who that person might be.

Many Republicans on both sides of the Capitol are doing a similarly delicate dance, trying to fend off the taint of Trump's more controversial statements while not launching a full-out assault on the current Republican front-runner.

Ryan's criticism was indirect but clear. "If a person wants to be a nominee of the Republican Party, there can be no evasion and no games," he said. "They must reject any group or cause that is built on bigotry. This party does not prey on people's prejudices."

Only one Republican senator, Ben Sasse of Nebraska, has come out firmly against Trump, saying he would not vote for him. Even so, Trump has picked up only five congressional endorsements, with the most prominent coming from Alabama Sen. Jeff Sessions.

A harsh new attack came from Rep. Scott Rigell, R-Virginia ,who is retiring at end of this year. Freed from the constraints of running for re-election, Rigell urged Virginians, who voted Tuesday in one of the 11 contests, to support anyone other than Trump, saying, "There is a con man among us."

"Trump is a bully, unworthy of our nomination," he wrote in an open letter on his Facebook page. "My love for our country eclipses my loyalty to our party, and to live with a clear conscience I will not support a nominee so lacking in the judgment, temperament and character needed to be our nation's commander-in-chief."

As the top Republican in the House, Ryan has largely tried to stay out of the presidential fray, routinely declining to comment on the race.

The last time he really took on Trump was late last year, after the real-estate mogul called for banning all Muslims from entering the U.S. That "is not what this party stands for, and more importantly, it's not what this country stands for," Ryan said at the time.

On Tuesday, speaking after a closed-door Republican conference, Ryan complained that the primaries should offer "the kind of moment where we should be having a serious debate about the policies needed to restore the American idea."

"Instead, the conversation over the last few days has been about white supremacist groups," Ryan said.

Trump has blamed a faulty television earpiece for his uncertain answers on Sunday to questions on CNN about Duke and the KKK, adding that he has previously distanced himself from Duke.

While he's been trying to stay out of the "day to day, ups and downs of the primary," Ryan said, "when I see something that runs counter to who we are as a party and as a country, I will speak up."

At the same time, Ryan insisted the party can weather its current strains.

"I never believe that our party is beyond the point of repair," he said. "Of course, we don't want to see divisions within our party."

He added that he hoped this would be the last time he has to speak out on the race.

He also deflected a question — stemming from Trump's suggestion — that Ryan's own stance as a vice presidential candidate in 2012 on changing entitlement programs hurt the chances of his ticket, headed by former Massachusetts Gov. Mitt Romney.

Ryan responded that he and Romney "won the senior vote" in 2012.

Asked about the prospect of supporting Trump if he emerged as the party's nominee, Ryan said, "My plan is to support the nominee."