This study in the Annals of Internal Medicine looked at women ages 55 through 69 years old, over the course of 19 years. Women who took multivitamins with B6, folic acid, magnesium, zinc or copper were at a 2.4 percent increased risk of death. Women who took vitamins with iron had a 3.9 percent increase.
The study couldn't explain why taking vitamin supplements could lead to an increased risk for cancer or heart disease, but researchers said they suspect taking too many vitamins over a period of time could eventually be toxic.
So what should women do?
Dr. Joanne Mortimer, director of the women's cancer program at City of Hope, said she would likely stop taking extra vitamins and minerals with the exception of calcium and vitamin D.
"Taking calcium was helpful… from a bone health standpoint," she said. "There also is data that says if you're vitamin D is low, and you correct that, taking vitamin D will help your bone health. But all the rest of these vitamins that we take, we probably get enough in our normal, healthy American diet and there probably is no reason to take additional supplements."
Mortimer said women should get their blood levels checked before they start taking vitamin D so they don't overdo it.
Vitamin D helps with the absorption of calcium. Besides these two supplements, Mortimer said the other important factors to promoting health are to maintain a normal weight and exercise regularly.
Mortimer said the study does not provide 100 percent convincing data that vitamins cause harm, but it clearly it does not show benefit from taking vitamins either.
The makers of dietary supplements said the report is biased and overstated concerns for elderly women.
People in the U.S. spend nearly $27 billion a year on supplements. More than half of adults in the U.S. take multivitamins daily.
ABC News contributed to this story.