"The vast majority of people in the United States have no interest whatsoever in street battles between the alt-right (better described today in more poignant terms) and the counter-protesters," libertarian essayist Jeffrey Tucker wrote on Saturday,

He added, “Most people have normal problems like paying bills, dealing with kids, getting health care, keeping life together under all the usual strains and mostly want these weird people to go away.”

It’s true.

We do not live in an America where neo-Nazis lurk around every corner, despite the unsettling visibility of some of these cretins over the weekend. While many of the counter-protesters rightly pushed back against white nationalists, some of them, too, are indefensibly violent, illiberal extremists, all too eager to throw the first punch.

But despite the chaos in Charlottesville and the tragic killing of one counter-protester by an alleged alt-right terrorist, I was more concerned, in the long-term at least, by some observers’ reactions on social media.

I saw some progressives blaming anyone who voted for Trump for the gathering of white nationalists, with plenty of other liberals Facebook liking or retweeting those sentiments, as if their most dire warnings about the Trump era had finally been vindicated. I saw conservatives try to equate progressives with the left extremist groups known as “antifa,” as if everyone who voted for Hillary Clinton, or, perhaps more accurately, Bernie Sanders, were no different than window smashers.

Both views are not only incorrect, but worse, harmful.

Many outlets described the racists convening as the “largest gathering” of white supremacists in decades. Reports estimated their numbers Saturday to be between 1,000 and 1,500. That’s a lot of open racists in one place — no hoods for most of them, unlike past eras — but not that many people, relatively. More importantly, I know many Trump voters, but not a single Adolf Hitler admirer.

How many Antifa members were in Charlottesville? How many are there, period? No one knows for sure, but among the Bernie Bros and “I’m With Her” liberals I know, I can’t think of a single one who has a face handkerchief and black hood on standby for face-smashing.

The few thousand who descended on Charlottesville over the weekend had a national spotlight, but they are an infinitesimally small group compared to a nation of 300 million. Yet how many people in the last 48 hours either equated, or were tempted to equate, those extremists on both sides with Americans they merely disagree with politically? And worse, in doing so, doesn’t that make them more like the extremists?

Despite whatever insecurities or mental disorders white nationalists might have, what ultimately drives them is a hatred for “the other.” Tucker’s essay observes:

The implausibility of their ideas is disguised by group psychology. They hang around people who think these same things and egg each other on in shared resentments … They conjure up scapegoats (blacks, Jews, women, Antifa, gays, and a government that is supposedly giving them all privileges at their expense) and begin to believe that the only way forward is to destroy them all in some grand uprising, after which they will seize power and rule forever.

Racists gain identity and purpose by hating people unlike themselves. How many Democrats loathe Trump voters and lump them all together, seeing them as “the other?” How many Trump fans do the same with liberals, dismissing the lot of them?

How many used the spectacle in Charlottesville to indulge in these political prejudices, which ultimately helps extremists exaggerate their size and influence? Racists want counter-protesters to show up. That’s the entire point. Antifa would be happy to fight neo-Nazis every day, if they could.

What was badly needed over the weekend was leadership, and Donald Trump failed miserably. The president clumsily and embarrassingly failed to condemn white nationalism over the weekend, and his “all sides” comment did no one any favors — with the possible exception of white nationalists.

Presidents are supposed to bring the country together during events such as these, and Trump doesn’t seem to grasp this yet (UPDATE: Trump finally condemned the white nationalists in strong terms, however late.)

Americans might be divided more right now than they were before the 2016 election, but they are certainly not polarized to the extreme degree on display in Charlottesville, Va., on Saturday.

Most Americans are not racists or leftist thugs. Extremists would no doubt like to frame the national debate within such parameters — wanting as many people as possible to believe they are the new mainstream — but these groups have the power to make that kind of mark only to the degree that we cede it to them.

Let’s not do that.