Sugiyama said it had been raining hard in the area all morning.
Self-defence forces will join firefighters and police in the rescue operation, he added.
Evacuation warnings were issued for a widespread area. The landslides appeared to have struck multiple times, about as fast as a car.
Footage showed a powerful, black mudslide move down a mountainside, knocking down and crushing houses and sweeping away cars in its way.
Helpless neighbours watched in horror, some recording on their phones.
NHK TV footage showed a part of a bridge had collapsed.
Atami is a quaint seaside resort area in Shizuoka prefecture, about 100km southwest of Tokyo.
The area that was hit by the mudslide, Izusan, includes hot springs, residential areas, shopping streets and a famous shrine.