I’ve lived 55 years in the South and I grew up liking the Confederate flag. I haven’t flown one for many decades, but for a reason that might surprise you.

I know the South well. We lived wherever the Marine Corps stationed my father: Georgia, Virginia, the Carolinas. As a child, my favorite uncle wasn’t in the military, but he did pack a .45 caliber Thompson submachine gun in his trunk. He was a leader in the Ku Klux Klan. Despite my role models, as a kid I was an inept racist. I got in trouble once in the first grade for calling a classmate the N-word. But he was Hispanic.

As I grew up and acquired the strange sensation called empathy (strange for boys, anyway), I learned that for black folks, the flutter of that flag felt like a poke in the eye with a sharp stick. And for the most prideful flag wavers, clearly that response was the point. I mean, come on. It’s a battle flag.

What the flag symbolizes for blacks is enough reason to take it down. But there’s another reason white Southerners shouldn’t fly it — or sport it on our state-issued license plates as some do here in North Carolina. The Confederacy — and the slavery that spawned it — was also one big con job on the Southern, white working class. A con job funded by some of the antebellum one-percenters, that continues today in a similar form.

You don’t have to be an economist to see that forcing blacks — a third of the South’s laborers — to work without pay drove down wages for everyone else. And not just in agriculture. A quarter of enslaved blacks worked in the construction, manufacturing and lumbering trades, cutting wages even for skilled white workers.

Thanks to the profitability of this no-wage/low-wage combination, a majority of American one-percenters were Southerners. Slavery made Southern states the richest in the country. The South was richer than any other country except England. But that vast wealth was invisible outside the plantation ballrooms. With low wages and few schools, Southern whites suffered a much lower land-ownership rate and a far lower literacy rate than northern whites.

My ancestor Canna Hyman and his two sons did own land and fought under that flag. A note from our family history says, “Someone came for them while they were plowing one day. They put their horses up and all three went away to the War and only one son, William, came back.”

Like Canna, most Southerners didn’t own slaves. But they were persuaded to risk their lives and limbs for the right of a few to get rich as Croesus from slavery. For their sacrifices and their votes, they earned two things before and after the Civil War: a skinny slice of the immense Southern pie and the shallow satisfaction of knowing blacks had no slice at all.

The plantation owners managed this con job partly with a propaganda technique familiar to modern Americans, but which hasn’t received the coverage it deserves in our sesquicentennial celebrations.

Starting in the 1840s, wealthy Southerners supported more than 30 regional pro-slavery magazines, many pamphlets and novels that falsely touted slave ownership as having benefits that would — in today’s lingo — trickle down to benefit non-slave-owning whites and even blacks. The flip side of the coin of this old-is-new trickle-down propaganda is the mistaken notion that any gains by blacks in wages, schools or health care would come at the expense of the white working class.

Today’s version of this con job no longer supports slavery, but it still works in the South and thrives in pro-trickle-down think tanks and media. These sources are underwritten by pro trickle-down one-percenters like the Koch brothers and Rupert Murdoch.

For example, a map of states that didn’t expand Medicaid — which would actually be a boon mostly to poor whites — resembles a map of the old Confederacy with a few other poor rural states thrown in. Another indication this divisive propaganda works on Southern whites came in 2012. Romney and Obama evenly split the white working class in the West, Midwest and Northeast. But in the South, we went 2-1 for Romney.

Lowering the flag because of the harm done to blacks is the right thing to do. We also need to lower it because it symbolizes the material harm the Confederate ideology did to Southern whites that lasts even to this day.

One can love the South without flying the battle flag. But it won’t help to get rid of an old symbol if we can’t also rid ourselves of the self-destructive beliefs that go with it. Only by shedding those too, will Southern whites finally catch up to the rest of the country in wages, health and education.

Frank Hyman lives in Durham, N.C. He’s a carpenter and stonemason and policy analyst for Southern Working Class Political Consulting.