One is being allowed to retire, another has been proposed for removal for cause, while a third has resigned.
The other eight agents are still on administrative leave, and their security clearances remain suspended. ABC News has learned that the agents involved will be forced to take lie detector tests.
Secret Service has been investigating charges that the agents brought prostitutes back to their hotel in Cartagena, Colombia, last week. The allegations, which also included excessive drinking, were serious enough for the Secret Service to pull the entire unit out Thursday before the president arrived. Another team was flown in to replace them.
The Secret Service is also investigating possible drug use by the agents.