The fire broke out at 3 a.m. on East Lambert Road which was shut down between State College Boulevard and Cliffwood Avenue.
The Brea Community Center was being used as a shelter for residents while crews worked on extinguishing the fire.
There were no reports of major injuries or of anyone being taken to the hospital, but four people were evaluated at the scene.
Officials said there was significant damage to the building. Residents were allowed to return home shortly after 7 a.m.