The incident happened around 8 p.m. Sunday near Baldwin Park and Ramona boulevards in front of the city's public library.
Firefighters say someone purposely tossed burning newspaper into the official drop box. One passerby told Eyewitness News the box looked like a chimney with smoke billowing out of it.
RELATED: What you need to know about ballot drop boxes ahead of 2020 election
Arson investigators have not yet released information about how many ballots were destroyed. Mayor Manuel Lozano said he believes there were close to 100 ballots in the box at the time of the incident that were burnt or wet.
"People are frustrated across the country, it's no different in Baldwin Park," Lozano said. "The incident that happened...does send a very bad message, as it is, the frustration with the voting box and then this occurs."
Fire crews say they did their best to save as many ballots as possible.
The ballot will be sent to the registrar's office, officials say.
The incident comes with just over two weeks until Election Day.
One voter said he's worried because he placed his ballot in the drop box a few weeks ago.
"It makes me very mad because I've never seen it, I've never seen this," said John Rios. "I'm 80 years old, I've been voting since I was 19, I've never seen something like this."
Multiple voters were seen attempting to drop their ballots off at the location Monday morning. Officials have cordoned off the box with tape and place a sign stating "box out of order."
If you believe your ballot may have been in that box, you can find out how to track your ballot at ABC7.com/vote.
RELATED: California GOP defends illegal ballot boxes, plans to expand program