Secretary of State Alex Padilla and Attorney General Xavier Becerra, both Democrats, say the boxes are illegal. They have ordered the party to remove them by Thursday.
The controversy started over the weekend when the secretary of state's office and county registrar's received reports of suspicious ballot drop boxes in Los Angeles, Orange, Ventura and Fresno counties.
RELATED: California Republican Party placed illegal ballot drop boxes in LA, Orange and Fresno counties
California Republican Party General Counsel Tom Hiltachk said Wednesday the party will not remove the boxes. He says they comply with California's ballot harvesting law that allows people to collect ballots from voters and return them to county election offices to be counted.
However, Padilla says this doesn't qualify as ballot harvesting because the collection boxes are unmanned. Democrats say the unofficial ballot drop boxes are a form of voter suppression because voters are now confused about which drop boxes are legitimate and which are not.
The state has threatened to take legal action if the boxes are not removed.
RELATED: California GOP defends illegal ballot boxes, plans to expand program
If any ballots were found in the unauthorized drop boxes, those voters could be contacted to vote again.
To find early voting and vote-by-mail ballot drop off locations, visit https://caearlyvoting.sos.ca.gov/ and to make sure your vote is counted, visit https://wheresmyballot.sos.ca.gov/.
The Associated Press contributed to this report.