The blaze began after a small Cessna went down in Blackburn Canyon, just southeast of Bakersfield, at about 11:30 a.m. Sunday.
The fire, which was only 5 percent contained, has destroyed more than 4,700 acres and one home.
"It's been a very aggressive, very angry fire," Kern County fire department spokesman Cary Wright said.
People in the rugged communities of Old West Ranch and Owens Ranch were ordered to evacuate. A relief center for evacuees was opened at Jacobsen Middle School at 711 Anita Dr. in Tehachapi.
It was unclear how many people were on the plane that crashed, but authorities confirmed two people were dead. Their names have not been released.
More than 600 firefighters, 25 engines, 15 dozers, four helicopters and 11 air tankers were on scene battling dry, windy conditions, as well as heavy fuel that includes beetle-infested trees.
The fire forced some road closures including Tehachapi Willow Spring between Highline Road and Cameron Road, southbound Dennison Road at Highline Road and Water Canyon Road at Highline Road.
The Associated Press contributed to this report.
