Secret Service spokesman Ed Donovan told ABC News that the man, 41-year-old James Dirk Crudup, was taken into custody just steps from the gate by agents who guard the White House complex around the clock.
According to Donovan, the homeless intruder had been previously ordered by a judge to stay away from the White House.
Crudup is being charged with contempt of court, in addition to unlawful entry, Donovan said.
Crudup, dressed in a blue polo shirt and white shorts, could be seen murmuring words as he was led away by security personnel.
Officials said he had a backpack with him. A bomb squad was working to examine the contents of the bag.
President Barack Obama was believed to be in the White House at the time.
Pennsylvania Avenue was closed to pedestrians.
The Associated Press contributed to this report.
