Meanwhile, Barbara Walters criticized her co-hosts for leaving the stage during the live show. However, she did say that O'Reilly should make the distinction that extremists committed the terrorist act.
Both Goldberg and Behar eventually returned to the stage after O'Reilly said that "if anyone felt that I was demeaning all Muslims, I apologize."
The mosque and Islamic center is planned for lower Manhattan, two blocks north of where the World Trade Center once stood.
The Associated Press contributed to this report.
