Lunada Bay gained notoriety for being run by an aggressive group of surfers known as the Lunada Bay Boys.
Crews began demolishing an illegal beach fort and hangout spot for the surfers on Monday. The hangout was equipped with a wood bench, table, canopy and stone fire pit.
Some area residents said the fort was built more than 30 years ago and should have been left alone.
"It's just a wrong thing to do and I'm neutral," Tom Kampas of Palos Verdes Estates said. "I've never been hassled by the so called Bay Boys up here at all."
Other area residents said it was a good start toward ending the alleged harassment by the territorial surf gang.
"It will be nice to be able to come out here and not have to worry about something happening," Patrina Roach of Palos Verdes Estates said.
The Palos Verdes Estates City Council voted in July to have the fort destroyed after visitors complained the Lunada Bay Boys used violence and threats to keep outsiders away.
A class-action lawsuit was also filed against the group earlier in 2016. In October, the city council awarded a construction company $61,000 to complete the demolition.
The demolition project was expected to be complete by Thursday.