Election officials were expected to continue counting ballots through the night and full preliminary results were not expected before Wednesday evening.
The contest was marred by allegations from Ter-Petrosian that authorities had rigged the vote and harassed his supporters. He asserted he was the real victor and urged supporters to rally in the capital Wednesday to protest the vote count.
The allegations of fraud and threats of mass protests raised concerns over instability in the volatile country at the juncture of the energy-rich Caspian Sea region and southern Europe and bordering Iran.
Sarkisian and Ter-Petrosian - Armenia's first president after the Soviet collapse - were the two top contenders among nine contenders vying to lead the South Caucasus nation, where more than a quarter of its 3.2 million people live in poverty despite some economic progress in recent years.
Tension with Azerbaijan over the disputed Nagorno-Karabakh region was one of the main issues of the campaign. Armenia's government says the mostly ethnic Armenian territory should be recognized as a sovereign state, while Azerbaijan says it will never give the region up.
Kosovo's declaration of independence from Serbia on Sunday added an element of uncertainty for Armenians, many of whom see clear analogies between Kosovo and Nagorno-Karabakh - a mountainous region in Azerbaijan that has been under the control of ethnic Armenian forces since a 1994 cease-fire ended a bloody six-year conflict.
The two presidential candidates differ on how to handle Nagorno-Karabakh.
Sarkisian, a native of the region, appears less likely to compromise than Ter-Petrosian, who was forced to resign as president in 1998 after advocating concessions in the dispute.