The sheriff's department said the move is to expand mental health services for inmates.
"This is not about immigration, this is a business decision based on serving the residents of Orange County and our immediate needs," said Sheriff Don Barnes.
Barnes said the move will save the department roughly $80 million.
Currently, 670 ICE detainees are being held at the Musick Facility in Irvine and the Theo Lacy Facility in Orange.
Barnes said it will also allow them to expand mental health and drug rehabilitation services in the jails.
Over the last five years, the county has seen a 40-percent spike in mental health cases. Some of the services will include counseling, medication management and psychiatric inpatient care.
The county will also be expanding the number of beds available.
OCSD said the Musick Facility will suspend operations until 2022. Once finalized, ICE will have 120 days to transfer detainees.
Immigrant rights groups are calling for many to be released, concerned about where they'll go and legal representation.
"You've made your profit off of them, now you're going to ship them off out of the state. No. You made a commitment to detain the immigrant community here, you need to provide the legal services to get those folks out," said Roberto Herrera.
In a statement, a spokesperson for ICE said this will negatively impact local operations.
"ICE will have to depend on its national system of detention bed space to place those detainees in locations farther away reducing the opportunities for in person family visitation and attorney coordination," the statement said.
Barnes said he has been informed by ICE the detainees will be moved out of the state.