According to an investigation by the Treasury inspector general, ineffective management at the IRS allowed agents to improperly target tea party groups. The report mostly blames IRS supervisors in Washington who oversaw a group of specialists in Cincinnati who screened applications for tax exempt status.
The report also states that a top supervisor in Washington was aware that agents were acting improperly, but did not adequately supervise them afterwards, enabling the practice to continue for more than 18 months.
Documents obtained by ABC News show letters targeting tea party groups also came from two Southern California IRS branches. One letter came from the office in El Monte, while another came from the branch in Laguna Nigel.
One letter asked for donor information, even though applicants seeking tax exempt stat are not required to provide such detail.
"They want to know all your members, they want to know your speakers and their backgrounds. It's very invasive and it seems totally inappropriate," said Dr. Ann Coil of the Santa Ana Tea Party Patriots.
Coil says she didn't bother seeking tax exempt status when her Tea Party group formed in 2010. She said it was not just because of the detailed IRS questionnaires and long wait time that she saw other tea parties dealing with, but she worried about retaliation from others.
"There is just a culture of fear and intimidation when you are a conservative and if you speak out," Coil said.
The IRS has publicly acknowledged and apologized for what it calls "inappropriate" targeting of conservative groups.
President Barack Obama has called the scandal outrageous and wants anyone involved to be held accountable.
On Friday, Congress will hold the first of what will likely be many hearings on the scandal, as lawmakers demand answers.
According to the Treasury report, none of the IRS agents doing the targeting say they were ordered to do it by anyone outside of the IRS. The FBI investigation into the IRS will determine if any laws were broken. Wrongdoing could result in criminal charges and possibly up to 10 years in prison.
"I think, as everyone can agree, if not criminal, they were certainly outrageous and unacceptable, but we are examining the facts to see if there were criminal violations," Holder said.