Repeal advocates said the fight wasn't over and gay rights advocates were furious because the Senate vote failed mostly because of procedural disagreement.
More than 60 senators were expected to support repeal, with at least four Republicans having said they support overturning the policy.
However, GOP senators demanded that the Senate vote on tax cut firsts.
They also want to be given extensive time debate on the defense bill, which contained other divisive provisions including one that would allow abortions at overseas military facilities.
Democrats had said Thursday morning they remained hopeful a last-minute deal could be struck to advance the legislation.
The rejection was a defeat for President Barack Obama, who promised to overturn the ban and later called it one of his top legislative priorities. But in the end, the White House did little to push the legislation, focusing its influence instead on tax cuts and a nuclear arms treaty with Russia.
The Associated Press contributed to this story.
