Justice Anthony Kennedy, a native Californian and Reagan appointee, often is that swing vote.
With the 9th U.S. Circuit Court of Appeals ruling Tuesday that Proposition 8, California's voter-approved ban on same-sex marriage, is unconstitutional, experts say Kennedy could be the key if it reaches the high court.
Pacific McGeorge School of Law Professor Larry Levine teaches sexual orientation law and knows Justice Kennedy, who taught at the Sacramento campus for years and comes back once in a while to visit or lecture.
One Colorado case called "Romer" is similar to Prop. 8. It wasn't same-sex marriage, but an initiative denying gays and lesbian protection from things like job and housing discrimination.
Justice Kennedy was clear in saying that initiative was wrong in his 1996 decision.
"Justice Kennedy very powerfully said that you cannot under the equal protection clause single out a group (and say) we're going to take away rights, we're going to treat you differently, because we don't like you," Levine said.
The attorney for Prop. 8 supporters, Andy Pugno, says the Romer case doesn't apply.
"In this case, the people were just exercising their right to re-instate the definition of marriage," Pugno said.
Pugno also questions whether Prop. 8 will really come down to Justice Kennedy.
"Declaring a right to same-sex marriage that would sweep away the marriage laws of most states in the country would be a very radical step and I just don't see our Supreme Court doing that," Pugno said.
All the speculation on Justice Kennedy may be for nothing. The U.S. Supreme Court accepts only 1 percent of proposed cases each year.
The Prop. 8 ruling could be considered so narrowly tailored to California that it requires no review.