The tragic and horrific incident happened in the city of Elizabeth around 5:30 p.m.
Police say the child fell from a rear window and was mauled by the animals, which belonged to the family and were in a cluttered fenced in area.
Neighbor say it was an horrendous sight as the child's mother tried to pull the dogs off the little boy.
"I didn't sleep all night," neighbor Maria Rocha said. "My heart is broken. I didn't sleep all night."
He was rushed to Trinitas Regional Medical Center, where he was pronounced dead shortly after 6:30 p.m.
Area residents say the dogs were always a terror, and the yard where the vicious attack happened is full of feces, motorcycles and debris.
"I cannot believe this happened to a little 3-year-old," Rocha said. "He's a baby. Could be mine, could be somebody else. He's a baby, and killed by dogs?"
Neighbors said their thoughts are with the boy who suffered such an agonizing death.
"I just hope that he didn't suffer, you know what I mean?" Rocha said. "I hope he didn't suffer, that it was quick."
Acting Union County Prosecutor Lyndsay Ruotolo, Elizabeth Police Chief Giacomo Sacca and Police Director Earl Graves said the circumstances are under investigation, but at this point, no charges have been filed.
Representatives of a local animal control organization responded to the scene and secured the dogs, which were transported to another location.
Anyone with any information about this matter is being urged to contact Prosecutor's Office Detective Richard Acosta at 908-347-0404 or Detective Michael Tambini at 908-472-4301.