Johnson spoke with ESPN's "First Take" Monday morning about the things that were happening behind the scenes that led to his decision.
"I started hearing, 'Magic, you're not working hard enough' and 'Magic is not in the office' and people around the Lakers office were telling me Rob (Pelinka) was saying things, and I didn't like those things being said behind my back," Johnson said.
He made it clear he was referring to General Manager Rob Pelinka the day he quit when he mentioned the backstabbing and whispering going on behind his back.
Johnson said he also didn't feel he had the power necessary to make the decisions.
"The straw that broke the camel's back was I wanted to fire Luke Walton," he explained. "I said: 'Listen, we gotta get a better coach.'"
Johnson said he was happier before this job and wants to go back to having fun and being able to help players around the league.
"Sometimes, as a man, you have to make decisions based on your own happiness," he said. "I could have done it in a different way. But I did it my way."
Johnson announced he was stepping down last month.