The agents allegedly engaged in inappropriate conduct involving prostitutes. Five special operations soldiers who were there to assist the Secret Service may have been involved as well.
The argument was a dispute over the bill for services rendered, the administration official said.
The scandal is an unwanted distraction for President Obama, who was attending what's billed as the Summit of the Americas. An American official tells ABC News the incident doesn't involve agents who guard the president every day. The personnel in question were working on security in advance of the president's visit to Colombia.
The allegations, which also included excessive drinking, were serious enough for the Secret Service to pull the entire unit out Thursday before the president arrived. Another team was flown in to replace them.
The agency said at no point was the president's security compromised.
In a statement, the Secret Service promised a thorough investigation.
"The Secret Service takes all allegations of misconduct seriously," the statement said. "This entire matter has been turned over to our office of professional responsibility, which serves as the agency's internal affairs component."
The last time the Secret Service faced this kind of public scrutiny was when agents allowed the now infamous party crashers, the Salahis, to attend a state dinner in November 2009.
