"All I heard was popping and a few shouts and I ran out. By the time I got out there it was completely engulfed in flame. Thank God that nobody was in there," said barn caretaker Charles Petery.
Authorities said the daughter of one of the owners, her husband and two young children live in the building. They were not home at the time.
"The roof has collapsed, we have partial collapse of the interior of the building, so there's been significant damage to the barn," said OC Fire Authority Capt. Mark Stone.
Authorities say the fire burned quickly due to a couple of factors.
"The old wood and the wind just provided for a large fire for us today," Stone said.
The red barn was built in 1898. Many residents remember it being used to hold dances, antique sales and other get-togethers.
"I think a lot of people are going to be very sad. It's just always been there, it's just always something you can count on standing there, and to see it go up like this is kind of traumatic," said San Juan Capistrano resident Nash Elliot.
The cause of the fire is still under investigation.
Fire officials say the family was in the process of converting the barn into a home. Investigators are trying to determine if the conversion may have played a factor in the fire.
Meanwhile the family, now without a home, is getting help from neighbors.
Residents are grateful no one was injured, but they mourn the loss of a landmark.
"You can't replace that. You can't replace the memories or all the stuff that goes along with that," said San Juan Capistrano resident Camille Woods.