At issue was what should be on the agenda for the next round of discussions.
There has been hope the two countries could hold high-level defense discussions. South Korea wants those defense talks to focus on two attacks by the North last year.
South Korea blames the North for a warship sinking last March that killed 46 sailors. North Korea denies any involvement in that attack and wants the agenda to include other issues.
The fact that the two could not decide on an agenda shows how difficult the relationship has become in the past few years, after South Korean President Lee Myung-bak's administration took a harder line with the North.
The Associated Press contributed to this report.
