Here's how Brown responded in an exclusive time magazine video posted on Time.com:
"Meg Whitman, she stops at nothing. She's even got Clinton lying about me. Did you see that? Where he said I raised taxes? It's a lie. I mean, Clinton's a nice guy, but whoever said he told the truth? You remember there's a whole story there about 'did he or didn't he?' OK, I did not have taxes with this state."
Brown called Clinton's office Monday to apologize after the video surfaced.
He also told reporters: "Bill Clinton was an excellent president. It was wrong for me to joke about an incident from many years ago, and I'm sorry."
The source of Clinton's criticism was a CNN report by Brooks Jackson, who issued a statement over the weekend on Factcheck.org saying he had mischaracterized Brown's record on taxes.
"Brown is right; I made a mistake in my 1992 report," Jackson wrote.
Specifically, Jackson said he picked the wrong year in concluding that California's state taxes were higher when Brown left office than in his first year.
However, Jackson defended his criticism of Brown's claim that he cut taxes as governor, noting that taxes were slashed by voters' approval of Proposition 13.
The Associated Press contributed to this report.