The 95-year-old is the father of Maria Shriver, who is married to former California Gov. Arnold Schwarzenegger.
Shriver announced in 2003 that he had Alzheimer's disease. His wife, Special Olympics founder Eunice Kennedy Shriver, died in 2009.
Shriver served as the first Peace Corps director in the administration of his brother-in-law, President John F. Kennedy. He also was Democrat George McGovern's running mate in 1972.
Shriver had fought for integration in Chicago and helped persuade Kennedy to make a crucial decision in the 1960 campaign despite other staffers' fears of a white backlash: When the Rev. Martin Luther King Jr. was jailed in Georgia that fall, Kennedy phoned King's wife and offered support. His gesture was deeply appreciated by King's family and brought the candidate crucial support.
Soon after taking office, President Kennedy named Shriver to fulfill a campaign promise to start the Peace Corps. Although it was belittled by some as a "kiddie corps," Shriver quickly built the agency into an international institution.
In 1994, Shriver received the Presidential Medal of Freedom, the nation's highest civilian honor, from President Bill Clinton.
The family of Robert Sargent Shriver, Jr. released the following statement:
Our dad, Robert Sargent Shriver, Jr., lovingly known as "Sarge," today went to heaven to join the love of his life, our mother, Eunice Kennedy Shriver. He was surrounded by his five children, five children in-law, and his 19 grandchildren.
He was a man of giant love, energy, enthusiasm, and commitment. He lived to make the world a more joyful, faithful, and compassionate place. He centered everything on his faith and his family. He worked on stages both large and small but in the end, he will be best known for his love of others. No one ever came into his presence without feeling his passion and his enthusiasm for them. He loved God, he loved Eunice, he loved us, he loved anyone who was a servant of peace, justice or joy. He loved life.
We pray that his spirit and example will guide us as we accept the challenge of living as he did. We will miss him forever. May the angels and saints and all his family receive him with a party beyond all imagining.
The Associated Press contributed to this report.