Holder made the announcement at an American Bar Association meeting in San Francisco Monday morning.
The attorney general said federal prisons are already operating at nearly 40 percent above capacity, and nearly half of all prisons are filled with inmates convicted of drug-related crimes. Holder said he also favors diverting people convicted of low-level offenses to drug treatment and community service programs and expanding a prison program to allow for release of some elderly, non-violent offenders.
"We need to ensure that incarceration is used to punish, deter and rehabilitate - not merely to convict, warehouse and forget," said Holder.
In one important change, the attorney general is altering Justice Department policy so that low-level, non-violent drug offenders with no ties to large-scale organizations, gangs or cartels won't be charged with offenses that impose mandatory minimum sentences.
Mandatory minimum prison sentences, a product of the government's war on drugs that began in the 1980s, limit the discretion of judges to impose shorter prison sentences.
Under the altered policy, the attorney general said defendants will instead be charged with offenses for which accompanying sentences "are better suited to their individual conduct, rather than excessive prison terms more appropriate for violent criminals or drug kingpins."
It's not yet known how each of the 94 U.S. Attorneys offices around the country will implement changes, given the authority of prosecutors to exercise discretion in how they handle their criminal cases.
Holder also said the department is expanding a policy for considering compassionate release for inmates facing extraordinary or compelling circumstances, and who pose no threat to the public. He said the expansion will include elderly inmates who did not commit violent crimes and who have served significant portions of their sentences.
The Associated Press contributed to this report.
