"We can only know what the records of the Secret Service reveal," Connecticut Sen. Joe Lieberman said in opening the first Senate hearing into the matter.
Lieberman said records show 64 instances of allegations or complaints of sexual misconduct made against Secret Service employees in the last five years. Many of the instances involved employees sending sexually suggestive emails. Three were about charges of inappropriate relationships with a foreign national and one was a complaint of non-consensual intercourse.
Secret Service Director Mark Sullivan apologized for the conduct of the employees in Colombia, and stressed that the agency does not condone such behavior.
Sullivan said the incident was not representative of the agency's nearly 7,000 employees, calling them "among the most dedicated, hardest working, self-sacrificing employees within the federal government."
Sullivan challenged a Washington Post story that said the Secret Service condoned risqué behavior - based on "numerous anonymous sources."
"If there is information out there, you know, when you read about it in the paper from an anonymous source, it's very difficult for us to investigate that type of an allegation," Sullivan said. "We would like to know who, when, where, and why, and names of people, and, you know, who are these people who are condoning it."
Republican Sen. Susan Collins of Maine told the hearing that the employees' actions in Colombia could have provided a foreign intelligence service, drug cartels or other criminals with opportunities for blackmail or coercion threatening the president's safety. And she challenged early assurances that the scandal appeared to be an isolated incident, noting that two participants were Secret Service supervisors - one with 21 years of service and the other with 22 years - and both were married.
"This was not a one-time event," said Collins, the senior Republican on the Senate Homeland Security and Governmental Affairs Committee. "The circumstances unfortunately suggest an issue of culture."
Lieberman agreed, saying, "It is hard for many people, including me, to believe that on one night in April 2012 in Cartagena, Colombia, 11 secret service agents - there to protect the president - suddenly and spontaneously did something they or other agents had never done before; that is to say, gone in groups of two, three or four to four different night clubs or strip joints and drank to excess and bring foreign national women back to their hotel rooms."
Homeland Security Secretary Janet Napolitano has testified previously that she believed the incident to be an isolated one, but senators have been skeptical.
Eight Secret Service employees have been forced out so far. Two of the ousted officers are fighting to keep their jobs.
Sullivan said the two originally resigned and are now fighting to get their jobs back. The agency will move to permanently revoke their security clearance, which is required for their jobs.
The Associated Press and ABC News contributed to this report.