/*Gadhafi*/ and his wife were inside the home in Tripoli when it was targeted, but they escaped unharmed.
Al-Arab, 29, and three of Gadhafi's grandchildren, all younger than 12, were killed, said a Libyan government spokesman.
The attack came hours after Gadhafi called for a cease fire and negotiations in what rebels called a "publicity stunt."
A Libyan government spokesman denounced the missile strike as a crime and violation of international law.
"This was a direct operation to assassinate the leader of this country. This is not permitted by international law. This is not permitted by any moral code," said spokesman Mussa Ibrahim.
NATO insists all of its targets are military in nature and linked to Gadhafi's systematic attacks on the population.
"It was not targeted against any individual," NATO spokeswoman Carmen Romero said Sunday, adding the report of the deaths remained unconfirmed.
British Prime Minister David Cameron also told the British Broadcasting Corp. that the strike was in line with the U.N. mandate to prevent "a loss of civilian life by targeting Gadhafi's war-making machine."
The incident also drew criticism from Russia, which accused the alliance of going beyond its U.N. mandate to protect Libyan civilians by trying to kill Gadhafi.
Gadhafi, who has been in power for more than four decades, has fought fiercely to put down an uprising against his regime that began with protests inspired by a wave of Mideast unrest and escalated into an armed rebellion. But the two sides have been locked in a stalemate, with Gadhafi holding much of the western half of the country and the rebels maintaining their eastern stronghold.
NATO warplanes have been carrying out airstrikes in Libya for the past month as part of a U.N. mandate to protect Libyan civilians.
The Associated Press contributed to this report.
