"It was just horrible. People getting crushed. Bodies, dead children, the whole thing was pretty awful," island resident Simon Prince told The Associated Press.
Some of those aboard were from Iraq, Home Minister Brendan O'Connor said from Christmas Island.
In Canberra, Immigration Minister Chris Bowen told Sky News that 28 people died and 44 have been rescued. Of those saved, 11 were under age 18.
Bowen said that up to 100 people could have been aboard the vessel, although O'Connor said the exact number is still unknown and a rescue effort would resume at dawn Thursday for any survivors.
Women and children were among the dead, Western Australia state Premier Colin Barnett said.
The Royal Flying Doctor Service sent doctors to the island to treat 30 injured victims, said Joeley Pettit-Scott, the group's spokeswoman. Three people were critically injured, two men with head injuries and one woman with blunt abdominal trauma, she said.
Photos and video from witnesses showed the boat crashing into jagged rocks and breaking apart, as well as people floating in the water amid the wreckage. The boat was about 20 to 30 feet (6 to 9 meters) long, with a cabin covered by a sheet of fabric or plastic.
Prince, who lives next to the cliff where the boat crashed, said he was awakened early Wednesday by what he thought were cheers. When he walked to the cliff, he determined that they were cries for help from a boat just offshore.
"The engine had failed," Prince told the AP. "They were washing backward and forward very close to the cliffs here, which are jagged limestone cliffs, very nasty."
Prince called police and soon there were dozens of locals standing on the cliff, wondering how they could help despite the storm and crashing waves. He said the boat tossed for an hour before it finally hit the rocks.
"When the boat hit the cliff, there was a sickening crack. All the people on board rushed to the land side, which is the worst thing they could do, but I don't think anybody could swim," he said.
Resident Michael Foster watched in horror as women and children screamed for help in the churning seas. "They had lifejackets on them, but the water was just pushing them up ... and throwing them towards the rocks," Foster said. "It was a pretty horrible situation."
Acting Prime Minister Wayne Swan said the vessel was a people smuggler's boat.
"This incident is a tragic reminder of the danger faced by people fleeing persecution and human rights violations in their home countries, and the desperate measures they will resort to in search of safety," said Richard Towle, the United Nations Refugee Agency's regional representative.
Melissa Fleming of the U.N. High Commissioner for Refugees expressed "deep shock and sorrow" at the loss of life.
"We are encouraged by reports that the quick rescue action saved the lives of as many as 41 passengers," she said. "But dozens more drowned."
Christmas Island is a remote Australian territory closer to Indonesia than the Australian mainland and a frequent target of refugee hopefuls, who are housed in a detention center there.
In recent years, many asylum seekers have come from Afghanistan, Iran, Iraq and Myanmar. Generally, they first fly to Indonesia and then continue on to Australia by sea.
International Organization for Migration spokesman Jean-Philippe Chauzy said asylum seekers pay $6,000-$7,000 to people smugglers, only to be told to pay another $1,000-$1,500 for the last stretch from Indonesia to Australia.
"Indonesia has been a stepping stone for economic migrants. ... Many become stranded in Indonesia when they run out of money, are cheated by people smugglers, or are intercepted by the Indonesian authorities," Chauzy said.
According the U.N. refugee agency, an estimated 848 people died or disappeared in 2009 in Italy, Yemen, Spain, and Greece - the main areas worldwide of large-scale migration.
"Climbing over razor wire fences, taking to sea in leaking boats or stowing away in airless containers, refugees and migrants around the world risk their lives every day in desperate attempts to find safety or a better life," the UNHCR said on its website.
One of the deadliest incidents in the region occurred on Oct. 19, 2001, when 374 people died - most of them believed to be asylum-seekers from Afghanistan and Iraq - in the sinking of a refugee boat en route from Indonesia to Australia.
In October 2009, a boat believed to be carrying about 100 asylum seekers from Indonesia to Australia disappeared and may have sunk, officials said.
In March of that same year, 237 people - most of them African migrants - drowned when a wooden vessel bound for Europe from Libya capsized in the Mediterranean.
That summer, a sailboat overloaded with Haitian migrants capsized off the Turks and Caicos Islands, and about 80 people were missing and presumed drowned.