Holbrooke had a central role in U.S. diplomacy for nearly five decades.
He is credited with negotiating the 1995 Bosnian peace agreement, ending that bloody conflict.
Holbrooke was nominated for a Nobel Peace Prize for his work.
Obama praised Holbrooke for making America safer.
"He is simply one of the giants of American foreign policy," Obama said Monday during a holiday reception at the State Department.
The secretary of state also commended Holbrooke for his long service when she visited him at George Washington University Hospital this weekend.
"He has given nearly 50 years of his life to serving the United States," Clinton said during a meeting in Canada.
Holbrooke received phone calls from the presidents of Afghanistan and Pakistan.
Holbrooke was born in New York in 1941 to Jewish immigrants. Right after graduating from Brown University in 1962, Holbrooke became a foreign service officer and was sent to Vietnam. He also worked as a magazine editor, professor, author and investment banker.
He said working nearly two years as Obama's special envoy to Afghanistan and Pakistan was the "Most difficult job I've had in my career."
Holbrooke is survived by his wife Kati Marton and her two children from her previous marriage to ABC anchorman Peter Jennings.
The Associated Press contributed to this report.