Video released by plant operator Tokyo Electric Power Co. shows how badly the nuclear plant was damaged in the March 11 earthquake and tsunami.
Officials said 25,000 tons of contaminated water have collected in the basement of Unit 2.
Removing it will allow access for workers trying to restore the plant's vital cooling system.
TEPCO hopes to reach a cold shutdown of the plant in about nine months.
But government officials acknowledge that setbacks could slow the timeline.
The water will be removed in stages, with the first third of it to be handled over the coming 20 days, said Hidehiko Nishiyama of Japan's Nuclear and Industrial Safety Agency. In all, there are 70,000 tons of contaminated water to be removed from the plant's reactor and turbine buildings and nearby trenches, and the entire process could take months.
With the nuclear crisis dragging on, public frustration with the government is growing.
TEPCO has offered residents forced to evacuate from homes around the plant about $12,000 per household as interim compensation.
The Associated Press contributed to this report.
