In the five weeks since he declared his support for a comprehensive immigration overhaul, U.S. Rep. Daniel Webster has gotten an earful.

One constituent told the second-term Republican that immigrants carry disease. Another said immigrants would steal jobs away from Americans.

“You cannot stop illegal immigration by rewarding it,” another man said at a recent town hall-style meeting in Groveland, a rural community west of Orlando. “Amnesty is a reward.”

As Congress returns to work this week after its summer break, Webster faces perhaps an even tougher crowd: fellow Republicans.

Webster is among about two dozen GOP lawmakers who support an eventual path to citizenship for millions of people who are living in the U.S. illegally. These Republicans are facing the daunting challenge of trying to persuade colleagues to follow them.

Most Republicans oppose this approach on citizenship, and there is little political incentive for them to change their minds. Only 24 of 233 Republicans represent districts where more than one-quarter of their constituents are Hispanic.

Even so, some in the Republican Party argue that its future hinges on whether the House finds a way to embrace an immigration overhaul, which is a crucial issue for the country’s fast-growing bloc of Hispanic voters.

Supporters of a path to citizenship point to demographic changes and business backing that have helped sway Webster, who for years opposed immigrant-rights legislation, as potential motives for wavering lawmakers to sign on.

“I think as a country we need to do something,” Webster said in an interview, echoing the rhetoric of Florida Sen. Marco Rubio and other prominent Republicans. “Doing nothing is amnesty.”

The small but growing band of Republicans is trying to strike a balance between conservative activists who want border security and immigration advocates who want a path to citizenship.

Many come from swing districts with sizable Hispanic populations that could make a difference in next year’s elections, tipping the balance of power in the GOP-controlled House. The lawmakers also feel the pressure from business interests that rely on immigrant labor.

At the same time, conservative taxpayer groups who typically fund GOP primary challenges have remained largely silent on immigration. Anti-immigration activists have failed to organize large-scale demonstrations or generate the kind of public backlash that killed Congress’ last attempt to remake immigration policy, in 2007.

Immigrant advocates, on the other hand, have waged a well-funded, aggressive campaign to push for the legislation.

“Congresspeople who may have been on the fence are realizing it’s safe to get in the water,” said Ana Navarro, a GOP strategist who led Hispanic outreach for Sen. John McCain’s presidential campaign in 2008. “There is safety in numbers.”

Some Republicans seem to have little choice.

U.S. Rep. Mike Coffman of Colorado won election in 2008 in a conservative district by campaigning against an immigration overhaul. But an unfavorable redrawing of his district after the 2010 census left him in Democratic-leaning territory that President Barack Obama won last year and where Hispanics make up nearly 20 percent of the population. He is now pushing for a “compassionate” approach to immigration.

U.S. Rep. Joe Heck of Nevada also has seen the Hispanic population grow in his swing district in suburban Las Vegas. Heck has said the path to citizenship outlined in bipartisan legislation passed by the Senate is “reasonable.” The state GOP gave him political cover by becoming the first in the country to endorse comprehensive immigration changes.

Political analysts said reluctant House members should take note of the country’s changing demographics.

According to research by Tom Wong, a political scientist at University of California, San Diego, who studies the politics of immigration, six House Republicans will see their margin of victory in last year’s election eclipsed in 2014 by the number of Hispanics and Asians who reach voting age. More than a dozen others, including Webster, will experience similar changes over the next decade.