During Willoughby's tenure as district attorney, he hasn't just been an administrator.
As recently as last month he was in the courtroom getting a conviction against a Wake County jailer accused of killing an inmate by throwing him on his head, but he didn't win them all.
"There are things that I could have done better," said Willoughby. "I lost some cases I would have liked to have had another shot at and won."
And when you are the district attorney in the capital you are also handed duties most district attorneys wouldn't want like prosecuting political corruption -- many times in his own Democratic party.
"I've had to investigate and sometimes prosecute people that were friends of mine and that's not easy," he said.
It also wasn't easy to decide to walk away from a career as the elected D.A. but Willoughby says he knew he didn't want to leave mid-term and have a temporary replacement appointed.
"It gives the voters and the people of this county a chance to choose their own district attorney without, perhaps, the political influence of some outside group," said Willoughby.
Willoughby says the legacy he is most proud of is seeing so many people go from the D.A.'s office to successful careers as lawyers, judges, and public officials, but he won't be endorsing anyone who decides to run for the D.A.'s office.
"I don't think they need me to tell them who to vote for," he said.
Willoughby says he will accept the will of the public.
He was appointed acting district attorney by then-Gov. Jim Hunt in 1983. He was first elected in 1986 and was sworn-in in 1987.
Willoughby does not have plans to run for anything else. He told ABC11 Friday that the DA's job was the only position he ever wanted in politics. Now that he's leaving that job, he's planning on working a few more years as a lawyer.