Duffy's announcement that he would resign on September 23 was not expected, as the Wisconsin Republican cited health problems with a baby due in October.
"With much prayer, I have decided that this is the right time for me to take a break from public service in order to be the support my wife, baby and family need right now," Duffy said in a statement.
Duffy's district in the north and west of Wisconsin is favored to stay in Republican hands, as it has trended more and more to the GOP in recent years.
While Democrats have so far only had three lawmakers decide not to run for re-election, some elections experts think more could be on the way, mainly because of age.
After a turnover of just 9 percent in 2004, every election has brought double-digit percentage change in the House; 94 new members won in the 2010 election - 91 new lawmakers arrived after the 2018 mid-terms.
Across the Capitol, four Senators have already announced they won't run for re-election - three of the four are Republicans.
Congress returns the week after Labor Day - we'll see if there is a rush of people deciding to move on.