ABC News has learned that the agents were partying at a Cartagena brothel called the Pley Club. Secret Service investigators plan to interview hotel employees and visit clubs in the area to find out what was going on.
Among those being investigated were some of the most elite Service Agents. Those in question include two supervisors, three counter-assault agents and the counter-sniper agents. Sources told ABC News that military officials now suspect 10 or more of their own personnel may have been involved in soliciting sex from prostitutes in addition to those 11 Secret Service agents under investigation.
A week before President Barack Obama arrived in Colombia for the Summit of the Americas last weekend, officials at the Hotel Caribe became irritated with loud partying in rooms occupied by the agents.
Sources said agents brought at least 11 local prostitutes to the hotel. Officials said it all may have remained a secret if not for a prostitute who became irate.
"There was a dispute the next morning when one of the women did not leave the room," said Rep. Peter King, chair of the House Homeland Security Committee. "Police came and she refused to leave until the Secret Service agents paid money that was owed to her."
The Secret Service personnel were ordered back to Washington before the president arrived. They've been suspended pending the results of the investigation and their security clearance was revoked.
"We let the boss down," said Gen. Martin Dempsey, chairman of the Joint Chiefs of Staff. "Nobody is talking about what's going on in Colombia other than this incident."
The director of the Secret Service, known for its discretion and discipline, called the incident "embarrassing."
Prostitution is legal in the area of Colombia where the hotel is located.
It appears that a congressional hearing on the matter is inevitable. Some wonder if there is a larger pattern when it comes to foreign trips.
"To assume that 11 people did something on a one-time basis is a little bit questionable," said Rep. Darrell Issa, chair of the House Oversight and Government Reform Committee. "If one person has a failure, a personal failure, that's one thing. But for 11 people to violate the basic security premise, tells us there's a problem larger."