In a 5-4 decision, the Supreme Court ourt said the reduction was "required by the Constitution" to correct longstanding violations of inmates' rights.
Justice Anthony Kennedy, a California native, wrote the majority opinion, including photos of severe overcrowding. The court's four Democratic appointees joined with Kennedy.
Kennedy said the state's prison system is so overcrowded it's caused "needless suffering and death" when inmates sought medical care.
The order calls for the release of up to 46,000 inmates, but since the state has already begun the transfer of some prisoners to county jails, the state will have to reduce the prison population by roughly 34,000 over two years.
The self-described nonprofit public interest law firm Prison Law Office sued over constitutional conditions more than two decades ago.
"We are really hopeful that this is going to be a step that is going to solve California's prison crowding crisis and solve the truly horrible conditions inside California prisons," said Rebekah Evenson, a Prison Law Office attorney.
Crime-victim groups are afraid California streets will become more dangerous if the state starts letting convicted felons go.
"It's a disaster. There's going to be sex offenders out. They're going to be letting kidnappers out. They're going to be letting a whole host of really bad people back into California without the resources to protect the good citizens of California," said Nina Salarno-Ashford, an executive board member of Crime Victims United.
But supporters of the release order say there will be no wholesale opening of the gates, that there are ways to reduce the prison population while protecting Californians.
The Drug Policy Alliance points out California incarcerates people for minor drug possession.
"It's ridiculous overspending," said Glenn Backes, spokesman for the Drug Policy Alliance. "That is part of the prison overcrowding crisis is the drug sentencing and other runaway sentencing inflation of the prior decades."
The Brown administration says it'll comply with the order, but Governor Jerry Brown himself used the ruling to push his "realignment" plan, which transfers about 30,000 prisoners to local jails. But Republicans are blocking the tax extensions to pay for it, saying county jail is not enough of a deterrent.
"That invites repetitive, continue-to-the-do-the-same thing, or escalating do-worse behavior," said state Assemblyman Jim Nielsen (R-Gerber), Assembly Budget Vice Chairman.
California corrections officials have two weeks to turn in a plan.
The Supreme Court's 5-4 decision represents the largest prison-release order in U.S. history.