Officials said all six of those with life-threatening injuries are adults being treated at San Francisco General Hospital. They included three men and three women between 20 and 60 years of age.
The other 14 victims were taken to other hospitals.
According to officials, the driver of the open air double-decker bus was driving erratically and speeding for at least two blocks before ending at the site.
Police said he hit a bicyclist, who is in critical condition, before hitting several vehicles and scaffolding which then fell on more cars and people. That scaffolding is part of construction at the site of the new Apple store.
PHOTOS: Several injuries reported after tour bus accident in SF
"According to eyewitnesses, they saw the bus, which is double-decker open air bus, driving erratically down Post Street," said San Francisco Fire Department spokesperson Mindy Talmadge. "In the process, a few vehicles were struck then the bus came to rest against some scaffolding which fell into the street."
First responders had to pull people out because they were pinned beneath the bus and scaffolding.
"It was like a bomb going off. It just kept going boom, boom, boom, boom, boom like it wasn't gonna stop," witness John Zimmer said.
Police say they will investigate to determine what caused the crash.
"We will look at everything," said Officer Albie Esparza. "We will look at any mechanical malfunctions, we will look at any DUI or alcohol or narcotics as standard operating procedures go for collisions."
Officials said some victims were treated at the scene but others, including the driver and the bicyclist, were taken to the hospital.
SF Fire Dept. says driver of bus was driving erratically for at least one block before hitting scaffolding. pic.twitter.com/RMGgMmunWC— Lyanne Melendez (@LyanneMelendez) November 14, 2015