Trump inauguration live updates: Donald Trump moves swiftly to implement his new agenda

The Bidens boarded a government plane to leave Washington and head to Santa Ynez, California, following Trump's inauguration.

"We're leaving office -- we're not leaving the fight," Biden said in his remarks to staffers at Joint Base Andrews before boarding.

"I've been doing this for 50 years, you're the best group of people," Biden said. "It's been the honor of my life to serve as your president."

Biden departed as Trump continued to speak to his supporters at Emancipation Hall.

After promising 'unity' in his inaugural address, Trump is reverting back to bulldozing political opponents like Nancy Pelosi and Liz Cheney.

"She's guilty as hell," he said of Pelosi over her response to the attack on the U.S. Capitol -- where he was speaking -- buy a pro-Trump mob.

During a speech in the Emancipation Hall, President Donald Trump aired grievances against Nancy Pelosi and Liz Cheney after pledging "unity" in his inaugural address.

He called Liz Cheney, a Republican who endorsed Kamala Harris over Trump because of his actions after the 2020 election, a "disaster."

"She's a crying lunatic and crying, crying Adam Kinzinger, he's a super crier," Trump said.

In more off-the-cuff remarks inside the Capitol's Emancipation Hall after his address, Trump spoke about people who have been convicted for their roles in the Jan. 6 attack on the Capitol as "hostages."

"I was going to talk about the J6 hostages," Trump said. "But you'll be happy because, you know, it's action not words that count."

Trump previously promised to pardon some Jan. 6 rioters on Day 1.

He also went after former Reps. Liz Cheney and Adam Kinzinger, two Republican members of the House Jan. 6 Committee who are outspoken critics of Trump because of his actions after the 2020 election.

"And I was going to talk about the things that Joe did today with the pardons of people that were, very, very guilty of very bad crimes, like the Unselect Committee of political thugs," Trump said.

Joe Biden, now the former president, left the Capitol on a Marine helicopter that will bring him to Joint Base Andrews for a final farewell to Washington.

The Bidens and Trumps walked together to the helicopter. Both couples were holding hands.

Biden smiled and appeared to laugh as he spoke to Trump. Jill Biden and Melania Trump hugged as they said goodbye.

This is an exit for Biden after five decades in politics.

Just before he left, Kamala Harris and Doug Emhoff entered their own motorcade. She is traveling home to Los Angeles, her office confirmed to ABC News.