Researchers at the University of Connecticut School of Medicine found the dryers may suck in bacteria from flushing toilets and spread it onto your hands.
The study shows bathrooms without hand dryers had just six pathogens compared to 254 in bathrooms with hand dryers.
Researchers also found the bacteria could also be spread to other parts of a building, not just onto your hands.
"Within a large building, potentially pathogenic bacteria including bacterial spores may travel between rooms," the study said.
The report is published in the journal of Applied and Environmental Microbiology.