Local lifeguards, the U.S. Coast Guard and other agencies responded around 10:30 a.m. following reports of an overturned vessel near the peninsula of Point Loma, according to the San Diego Fire-Rescue Department.
The original call was for a handful of people overboard but as rescuers arrived in boats and jet skis they quickly realized "it was going to be a bigger situation with more people," said San Diego Lifeguard Services Lt. Rick Romero.
"There are people in the water, drowning, getting sucked out the rip current there," he said.
When officials arrived on scene near the Cabrillo National Monument there was a "large debris field" of broken wood and other items in the choppy waters. The wreckage of the vessel slowly disintegrated in the ocean after hitting a reef.
Seven people were found in the water, including three who had drowned, said Romero. One person was rescued from a cliff and the others managed to make it to shore on their own, he said.
Officials initially said 30 people were found, but the Coast Guard later said a total of 29 people were accounted for, including one person in critical condition and the four deceased individuals.
The Coast Guard said crews were still searching for more, since there was no official record of how many people were actually on the boat.
Twenty-five people were transported to hospitals with "a wide variety of injuries" including hypothermia, Romero said. Most of the victims were able to walk themselves to ambulances, he said.
Officials said the vessel is larger than the typical open-top wooden panga-style boats often used by smugglers to bring people illegally to the U.S. from Mexico.
WATCH: Officials give update after boat capsizes off San Diego
"Every indication from our perspective was this was a smuggling vessel. We haven't confirmed their nationality," said Jeff Stephenson, a supervising agent with U.S. Border Patrol.
A man believed to be the operator of the 40-foot vessel was taken into custody as a "suspected smuggler."
It is not clear where the boat was headed, and no word on what caused the boat to overturn.
This happened in federal water jurisdiction, so federal officials will handle the investigation. San Diego is assisting federal authorities.
On Thursday, border officials intercepted a panga-type vessel traveling without navigation lights 11 miles off the coast of Point Loma with 21 people on board. The crew took all 15 men and six women into custody. Agents determined all were Mexican citizens with no legal status to enter the U.S., according to a statement released by Customs and Border Protection. Two of the people on the boat, the suspected smugglers, will face charges, it said.
The Associated Press contributed to this report.