The protesters called for the reopening of L.A. as new limits on gatherings, activities and business occupancy were set to take effect Monday.
Video from the scene reportedly showed a handful of protesters arguing with one of Ferrer's neighbors.
One waved a flag reading "Trump Train," and another held a homemade sign, saying, "Only we the people can save the USA."
The demonstration lasted about an hour and was reported to have broken up about 5 p.m.
Most of the demonstrators were not wearing masks, even as the county and country see record numbers of cases and hospitalizations.
A similar protest was also held outside a restaurant in Sherman Oaks.
Demonstrators outside of Casa Vega say the restaurant and hospitality industries have become scapegoats for the recent surge in cases, leading to over 700,000 lost jobs since the onset of the pandemic.
City News Service contributed to this report.