The vote came after dozens of veterans and community members blasted City Councilman Gregory Salcido and called for his resignation for the video in which he criticized the intelligence of those who join the military.
The council passed a resolution asking Salcido to resign from the council and removing him from city committees.
The resolution is non-binding and the council does not have the power to force his removal from the elected office.
Some members of the community, however, said they planned to launch a recall campaign.
Salcido also broke his public silence over the matter for this first time, standing by his comments, but saying he and his family have gotten death threats during the controversy.
"We've been inundated with death threats," he said. "People have told me they're gonna rape my wife, they're gonna kill me. They're gonna leave my son an orphan."
He said he would apologize for his comments if they hurt people and said that was not his intention. He said his intention was to motivate his students to go to college, rather than join the military. He repeated his contention that less-successful students more often go to the military.
"My goal as it relates to my students as it relates to the military is to get them to do everything to get to college. It's not just the military. I don't want them working at a fast-food restaurant either," he said. "I don't think it's a revelation to anybody that those who aren't stellar students usually find the military a better option. That's as plain as it's Tuesday night. That's not a criticism of anybody."
Salcido, a teacher at El Rancho High School, was recorded by a student delivering a speech against the military in the classroom.
In the recording, Salcido is heard using expletives about those who serve in the military and their intelligence: "Your freakin' stupid Uncle Louie or whatever. They're dumb- - -ts. They're not high-level thinkers. They're not academic people. They're not intellectual people. They're the freaking lowest of our low."
The recording sparked an outrage in the community and across the country.
Much of that anger was on display at Tuesday's council session.
"You disgust me," one woman told Salcido. "I was your friend. I stood by you. You disappointed me so much."
"Sir, I am ashamed that you would take the time to say these terrible things about the men and women who serve the people," said another man, an Army veteran. "Who serve in the armed forces, who serve in law enforcement. You're shameful. You should resign."
Salcido's fellow members on the council also denounced his remarks and said he should resign.
"I and my fellow council members strongly disagree with and denounce Mr. Salcido's comments regarding those men and women who have served or are currently serving our nation's military," Pico Rivera Mayor Gustavo Camacho said.
"We stand together in the belief that the majority of residents of our great city of Pico Rivera are grateful to those who have chosen to defend our country through service in our military," he added.
Councilman Brent Tercero said asking for his resignation was not enough because the council does not have the legal authority to actually force him to leave office.
He said he would volunteer to sign a recall petition and donate $250 to the effort.
"His comments do not reflect the beliefs and attitudes of the entire city council," Tercero said. "The comments were hurtful and insulting, not just to students hearing them but to the members of our community and even to councilmembers up here on the dais."