The home is designated as a sober living facility.
Residents are worried because the home is less than the required 2,000 feet from a local park, which is a violation of Megan's Kaw.
"One way in and out. They have to pass the park, which is just like passing a liquor store to an alcoholic," said Altadena resident Daniel Duncan.
"They're violent sexual offenders who have done incredible things against children. This is just not going to happen on our watch," said Altadena resident Latrell Cottle-Moore.
Residents are now trying to set up a meeting with the state parole board, which monitors parolees.
In the meantime, the sheriff's department says it will step up patrols in the area.