Researchers said if you're doing it the wrong way, you could be contributing to plastic pollution -- adding to the debris in rivers, lake and oceans.
MORE: FDA approves light-adaptive contact lenses that act as sunglasses
Scientists at Arizona State University found 15-20 percent of contact lens wearers flush their lenses down the sink or toilet.
"This is a pretty large number," said Charles Rolsky, an ASU Ph.D. student who presented results of a new nationwide study on Sunday.
Rolsky said considering about 45 million people in the U.S. alone wear contact lenses, that amounts to 1.8-3.36 billion lenses flushed per year.
MORE: How to avoid eye infections from contact lenses
"(That's) about 20-23 metric tons of wastewater-borne plastics annually," he said.
The researchers said the biodegradability of the lenses is unclear because of the unusual plastics used in them.
It's the first nationwide study on the environmental impact of contact lenses, a $2.7 billion industry in the U.S.
The ASU researchers are calling on contact lens manufacturers to provide product packaging information on how to safely dispose of the lenses, and for consumers to simply discard their lenses in the trash instead of putting them down the drain.