The museum posted the news on Twitter.
"Sunset at Montmajour" depicts trees, bushes and sky. The painting is an example of the artist's "transitional" work - a time when he increasingly adopted the thick brush strokes that would become typical of his work in the final years of his life, experts at the museum said.
It can be dated to the exact day it was painted because Van Gogh described it in a letter to his brother, Theo, and said he had painted it the previous day - July 4, 1888.
Museum director Axel Rueger unveiled the painting and described the discovery as a "once in a lifetime experience."
"This is a great painting from what many see as the high point of his artistic achievement, his period in Arles, in southern France," he said. "In the same period he painted works such as 'Sunflowers,' 'The Yellow House' and 'The Bedroom'."
The painting belongs to a private collector. "Sunset at Montmajour" will be on display at the museum later this month.
Officials said the museum had declined to authenticate the painting in 1991. Experts say the painting could have been declined for various reasons, including the fact that the painting is not signed by the artist.
But when the museum took a fresh look at the work in 2011, they had the advantage of a newly edited and published compendium of all Van Gogh's letters, and were able for the first time to identify the exact location "Sunset" depicts: Monmajour hill, near Arles, France. The ruins of Monmajour abbey can be seen in the background on the left side of the painting.
Van Gogh struggled with bouts of mental distress throughout his life, and died of a self-inflicted gun wound in 1890.
The Associated Press contributed to this report.
