The move came hours after Paterno, 84, announced that he planned to retire at the end of the season.
"I am disappointed with the Board of Trustees' decision, but I have to accept it," Paterno said in a statement. "A tragedy occurred, and we all have to have patience to let the legal process proceed."
Paterno went on to say that he is grateful to "all of the coaches, players and staff who have been a part of this program. And to all of our fans and supporters."
Thousands of Penn State students assembled on campus in support of Paterno immediately after the news of his firing broke.
Stunned and angered, many held signs and chanted his name as they took to the streets.
"I appreciate the outpouring of support but want to emphasize that everyone should remain calm and please respect the university, its property and all that we value," Paterno said.
As Paterno leaves his post as head coach, Tom Bradley will be the interim coach, according to ESPN.
Paterno has been called the greatest college football coach ever by fellow coaches. But now, the man who preached that his players achieve success and adhere to a moral code is forced out in disgrace over not doing enough over allegations of sexual abuse of young boys.
The U.S Education Department said it will investigate Penn State's handling of the child sexual abuse case.
Paterno's former assistant, Jerry Sandusky, was charged with molesting eight young boys between 1994 and 2009. Paterno said in a statement that he was "absolutely devastated" by the case.
"I grieve for the children and their families, and I pray for their comfort and relief," he said.
Earlier as he announced he was resigning, Paterno briefly talked to players in the auditorium of the Mildred and Louis Lasch Football Building. Standing at a podium, the coach told them he was leaving, then broke down in tears.
He was given a standing ovation when he walked out of the auditorium.
Paterno it out after his 46th season as head coach of the Nittany Lions. Paterno's players took the news especially hard.
"There's a lot of emotional people right now. We felt like maybe coach didn't get what he deserves," said Jack Crawford, a senior defensive end.
Though Paterno is not accused of any wrongdoing, he has been questioned over his apparent failure to follow up on an alleged incident in 2002 where he was told that Sandusky molested a 10-year-old boy in the Penn State showers. Paterno informed university officials, who apparently did nothing.
Penn State's Board of Trustees said it will appoint a special committee on Friday to examine the scandal.
ESPN analyst Stephen A. Smith had said Paterno should be removed immediately and should not be allowed to coach this Saturday's game.
"We're watching this game Saturday between Penn State and Nebraska. Nobody's going to be thinking about those games. What they're going to be thinking about is Joe Paterno on that sideline, and they're going to be thinking about child molestation while we are watching an NCAA football game, that is simply an unacceptable environment," Smith said.
Also under fire for not telling authorities about Sandusky is now former Penn State President Graham Spanier.
"I would never hesitate to report a crime if I had any suspicion that one had been committed," Spanier said in a statement released after his ouster.
"Our great university has been rocked by serious charges against a former coach," he said. "I was stunned and outraged to learn that any predatory act might have occurred in a University facility or by someone associated with the University."
"I have urged the board to deal with this quickly and swiftly. Other comments have to wait until after the board meetings," said Gov. Tom Corbett.
ABC News has learned more alleged victims have come forward after Sandusky was charged. Sandusky remains free on $100,000 bail. His preliminary hearing has been moved to next month.
The decision to fire the man affectionately known as "Joe Pa" brings to an end one of the most storied coaching careers, not just in college football, but in all sports. Paterno won 409 games, a record for major college football, and is in the middle of his 46th year as coach.
Read Joe Paterno and Graham Spanier's full statements
The Associated Press contributed to this report.
