At least two people were feared dead on Saturday after landslides triggered by heavy rains hit the central Japanese city of Atami, where about 20 people were still missing, public broadcaster NHK said.
Prime Minister Yoshihide Suga, who convened an emergency task force to tackle the crisis, asked people in the affected areas to remain on alert.
"There may be more heavy rainfalls and we need to be taking the highest caution," Suga said in televised remarks.
The floods are a reminder of the natural disasters - including earthquakes, volcanoes and tsunami - that plague Japan, where the capital Tokyo is to host the summer Olympics starting this month.
Two people were found in a state of cardiac arrest in Atami, 90 km (60 miles) southwest of Tokyo, and taken to hospital, NHK said.
It aired footage of collapsed and half-submerged houses as water dashed the city with mud and debris. Social media images showed partially submerged cars and rescue workers wading through waist-high water with a small life raft.
Japan's military sent emergency rescuers to the city, where about 80 people are evacuated, NHK said.
Some 2,830 households in the area face power outages, it said, citing the Tokyo Electric Power.