Neighbors said the fire may have been lit by a cigarette that was carelessly tossed.
"There were mattresses around there, and so, probably that started the fire," said neighbor Arianna Mendieta.
The mattresses were apparently put outside due to a bed-bug infestation.
The L.A. City Fire Department said it could not confirm that a cigarette sparked the blaze.
Mendieta said she helped her own family get out after she awoke to flames just feet from her bedroom window.
"I saw light outside. It was dark, so you know, the light stood out," she said. "I was scared because I had never seen a fire like this, so close to my house."
The fire was declared a knockdown at 1:03 a.m. Two rooms were gutted in the house.
There were nine men and five women who were evacuated, which includes two staff members. The Red Cross is helping to arrange places for the displaced residents to stay.
The owner of the home for the mentally challenged said she was not sure when the residents would be allowed to return.
No injuries were reported. The official cause of the fire has not been determined.
