The fireworks came amid a heated argument over the proposed Islamic center near the site of the Sept. 11 attacks.
"Listen to me," O'Reilly said, "because you'll learn something."
"Pinhead!" retorted "View" host Joy Behar, who also hosts a prime-time show on HLN.
O'Reilly, who was on the popular daytime talk show to promote his book "Pinheads and Patriots: Where You Stand in the Age of Obama," said locating the Islamic Center near ground zero is inappropriate "because Muslims killed us on 9/11."
Seconds later Behar and Goldberg walked off the stage.
"I don't want to sit here," Behar said. "I don't. I'm outraged by that statement."
The show's creator, Barbara Walters, was not happy. She immediately said her colleagues were wrong to walk off.
O'Reilly eventually apologized for his choice of words and the two co-hosts returned back to the set. But later on their respective prime time shows the war of words between Behar and O'Reilly continued.
This isn't the first time the ladies of "The View" have made headlines with controversy on their show and it likely won't be their last.
The feud was not discussed on the popular ABC show Friday morning because the episode was pre-taped. But the hosts say they will be addressing the controversy on their show on Monday.
The Associated Press contributed to this story.