"The misconduct that (the report) uncovered is inexcusable. It's inexcusable, and Americans are right to be angry about it, and I am angry about it," Mr. Obama said. "I will not tolerate this kind of behavior in any agency, but especially in the IRS given the power that it has and the reach that it has in all of our lives."
A Treasury Department report cited ineffective management for the agency's actions that signaled out Tea Party groups for extra scrutiny. The groups were seeking tax exempt status with the IRS.
According to the report, the practice started in 2010 and continued until last year. The IRS has publicly acknowledged and apologized for what it calls "inappropriate" targeting of conservative groups, and acting IRS Commissioner Steve Miller is out of a job.
"It's important to institute new leadership that can help restore confidence going forward. Second, we're going to put in place new safeguards to make sure this type of behavior cannot happen again," the president said.
Some members of Congress think the roots of the problem are much bigger than slack management practices and are calling for jail time for whoever was behind the policy.
Appearing before the House Judiciary Committee, Attorney General Holder is promising the Justice Department will take whatever action is necessary against those responsible and will also look at ways to prevent abuse in the future.
"To the extent that there are enforcement gaps that we find, we will let this committee know and hopefully work with this committee to make sure what happened ... does not happen again," Holder said.
Holder has ordered the FBI to investigate the agency's practices and policies in dealing with tea party political organizations. Other potential crimes include making false statements to authorities and violating the Hatch Act, which prohibits federal employees from engaging in some partisan political activities, Holder said.
"I can assure you and the American people that we will take a dispassionate view of this," Holder told the House Judiciary Committee at a hearing Wednesday. "This will not be about parties, this will not be about ideological persuasions. Anybody who has broken the law will be held accountable."
Holder engaged in heated exchanges with the Republican delegation during the hearing, telling Rep. Darrell Issa that it was "unacceptable" and "shameful" for him to "misrepresent the facts."
The Associated Press contributed to this report.