CHARLOTTESVILLE, Va. — One of the joys of growing up, for Deborah Bell-Burks, was sitting on the counter of the general store that her family owned on Main Street, helping her mother bag items and count change. A bicycle ride through the neighborhood would take her past a grocery, a barbershop and a cobbler’s shop. And most of the people she would see, well-off business owners and customers alike, were black like her, offering an early lesson in what she could become.

That was Vinegar Hill, this city’s thriving black cultural and economic hub, just west of downtown. But then came the urban renewal policies of the 1960s.

In a time when a poll tax kept many black residents from casting ballots, Charlottesville held a referendum and decided, by a margin of just 36 votes, to raze Vinegar Hill for redevelopment. Hundreds of residents and 29 black-owned businesses were displaced, including the store owned by Ms. Bell-Burks’s family. Most never reopened, and generations of black families have been trapped in public housing complexes since then, or have left town altogether.

“There was a lost opportunity for the development of African-American wealth in Charlottesville,” said Ms. Bell-Burks, 62.

As this city in central Virginia has become the latest flash point in America’s fight over white supremacy, a deep-rooted struggle for equality has weighed heavily on Charlottesville’s shrinking black population. And that struggle is bound up in some ways with the controversy over the city’s Confederate monuments.

The same day it voted to remove a statue of Robert E. Lee from a park — a move that white supremacists descended on the city to protest — the City Council did something that got much less publicity. It unanimously approved a $4 million spending plan to address racial disparities. Over the next five years, about $2.5 million is to be used to redevelop public housing; $250,000 will go to expanding a park in a black neighborhood; and $20,000 a year will pay for G.E.D. classes for public housing residents.

Activists call it reparations for the destruction of Vinegar Hill and other black neighborhoods here.

“I’m hoping that other elected officials and policy makers from across the country can see it’s not enough to just move a damn statue,” said Wes Bellamy, the vice mayor, who proposed the plan. “I think symbols matter. But you move the statue, and then what else? It’s cool to, quote-unquote, move the statues. But don’t try to pacify black folks or people of color.”

For some in Charlottesville, the controversy over the Confederate monuments is about more than what they represent: It is about the soul of the city.

Gentrification has been creeping through this university town for decades, while the black share of the population has fallen to 19 percent, from 22 percent in 2000.

The statues of Lee and Stonewall Jackson were erected in the 1920s, a time of strong white resistance to black demands for equality, and they stamped certain parts of town as the domain of white people, according to Marlon B. Ross, a professor of English at the University of Virginia, whose historic campus is on the city’s west side. According to an article in Slate written by three of his students, the Jackson statue was erected on land the county had confiscated seven years earlier from black owners.

“It sort of consolidates the landscape as white supremacist,” said K. Ian Grandison, a University of Virginia English professor who teaches a class on race and development with Professor Ross. “When these monuments were put there, there were big Confederate reunions, and they marched through these streets, just as is happening today — because they were basically saying, ‘You may have your 13th, 14th, 15th and 16th Amendments, but this is the Jim Crow era.”

Building is booming in Charlottesville. And while the hotels, boutique restaurants and luxury condominiums that are going up are not Confederate symbols, many black residents say the developments still feel like markers of exclusion, built for people who do not look like them.

“Nothing over there for me,” said Marie, 32, pointing to a blue crane hovering near the concrete skeleton of a building climbing to the skyline.

Marie, who asked that her surname not be published out of concern that being identified could hurt her future job prospects, was standing in the Westhaven public housing complex. It was built when Vinegar Hill was demolished, and it was where many families displaced from that area were forced to find new homes. Marie, who grew up in Westhaven, was braiding someone’s hair in front of the brick townhouse-style apartment where her mother still lives.

The complex, separated by railroad tracks and a grassy hill from the bustling restaurants and clothing boutiques on Main Street near the University of Virginia campus, sits sunk in a bowl of low ground with curving, chopped-up streets. It feels like a different world from the one above, where the new high-rise is being built and where a luxury condo peeks down over the far end of the complex. And it is one whose days, Marie said, seem to be numbered.

“They’re building that for the college students; they’re building that for the people migrating here,” said Marie, who works in a university dining hall. “When you all finish with that, you’ll want more. This is next. This is a cash commodity.”

Charlottesville, like many cities across the country, faces an affordable housing crisis. The city is one of the most expensive in Virginia for rental housing, and the median rent here has risen by 3.4 percent in the past year, according to data compiled by The Daily Progress. Nearly half of the city’s residents are considered to be burdened by high housing costs.

There are about 1,100 families on the waiting list for the city’s 376 public housing units, and some have been waiting for up to nine years, according to Grant Duffield, the executive director of the Charlottesville Redevelopment and Housing Authority. The city’s goals for expanding the supply are modest: It hopes to add 15 percent more affordable units by 2025.

The problem falls disproportionately on the city’s black residents, and resolving the racial disparity in living conditions will take more than an economic solution. There remain deeply ingrained racial and social divides that can be traced to the way the city was developed.

After Vinegar Hill was demolished, the land was vacant for several years, separated from downtown by wide new thoroughfares. Today, parking lots and large buildings, including a hotel, a courthouse and a cultural center, wall off pockets of poor black residents from the more upscale downtown pedestrian mall.

Just south of the mall, down the street from where a white supremacist’s car plowed into a crowd and killed Heather Heyer, is the last remnant of predominantly black affordable housing in the center of town: the Friendship Court Apartments, with nearly 150 units, subsidized with Section 8 federal housing money.

Some residents wonder whether it, too, could succumb to gentrification. The Piedmont Housing Alliance, a nonprofit group that owns the property, wants to redevelop it into a complex with 600 units. Preliminary plans call for keeping the 150 Section 8 units and adding some that would also be subsidized and income-restricted to some degree, but leasing most of the complex at market rates to more affluent tenants. Fears of displacement and social upheaval abound.

“There’s a concern about the dynamics,” said Tamara Wright, 38, a Friendship Court resident for about 10 years. “It’s like, ‘O.K., what if the rich people don’t like our kids?’ It’s just a general concern about whose concerns are going to seem greater or whose concerns are going to be met and seem like most important.”

Frank Stoner, the Piedmont board’s chairman, said that white people in Charlottesville still had work to do to earn the trust of black residents, who remember Vinegar Hill and the unkept promises of “something better for you.”

“How do we do this in a way that, from a social equity perspective, is comfortable and catalytic toward improved race relations?” said Mr. Stoner, who is white. “I think that is our primary focus.”