Despite warning taxpayers last year of a telephone scam that convinces victims they owe money to the Internal Revenue Service, investigators at the IRS say the reports are only growing of people losing money to callers who are claiming to represent the tax agency.

"The increasing number of people receiving these unsolicited calls from individuals who fraudently claim to represent the IRS is alarming," said IRS Inspector General J. Russell George in a statement.

"This is the largest scam of its kind that we have ever seen," George added, saying his office has received reports of "thousands of victims who have collectively paid over $1 million" to criminals - not the IRS.

In a news release issued on Thursday, the IRS took pains to note that the tax agency "usually first contacts people by mail - not by phone - about unpaid taxes."

"If someone unexpectedly calls claiming to be from the IRS and uses threatening language if you don't pay immediately, that is a sign that ir really isn't the IRS calling," warned George.

The IRS is even urging taxpayers to forward e-mails that look like they are tax phising scams to phishing@irs.gov.

Latest tax return statistics

The latest numbers from the IRS on this year's tax filing season continue to show the tax agency is more speedily processing tax returns, up almost 6 percent from the same point a year ago.

Self-prepared tax returns are up almost 6 percent as well, while returns completed by tax professions are down almost 2 percent.

One very interesting statistic is on how many taxpayers are using the IRS website - visits to IRS.gov are down over 10 percent - almost 23 million visits - from the same time in 2013.

It's not clear why taxpayers are avoiding irs.gov this year.

Some people still like to get their money the old-fashioned way, as 52.7 million direct deposit refunds have been authorized so far in 2014, up 0.7 percent.

About 9 million refunds were made in the form of a check.

More than 75 million tax returns have been received by the IRS as of last week - that's a bit over half of what the feds expect to get for this tax filing season.