Stamford Mayor Michael Pavia stressed that the deadly fire was a tragic accident and was not the result of foul play.
Neighbors said they were awakened by screams shortly before 5 a.m. Sunday and rushed outside to help but could do nothing as flames devoured the large Victorian home.
Two people made it out alive - well-known ad exec Madonna Badger and a male acquaintance. Authorities say the male acquaintance was a contractor who had done work on the home.
Police identified the victims as Badger's three children - a 10-year-old and 7-year-old twins - and both of her parents, who were visiting for the holidays.
Pavia said Badger's father, Lomer Johnson, was found outside on the roof of a small porch off a bedroom.
"It appears that he either was trying to get to his granddaughter from the outside or that he was leading his granddaughter out," he said.
Neighbors say the house was under renovation, and the family had just moved in a couple of months ago.
The severely damaged Victorian house was torn down Monday after the buildings department determined it was unsafe and ordered it razed, Stamford fire Chief Antonio Conte said.
The Associated Press contributed to this report.
