Hagiuda acknowledged that there is little data on whether fathers are unfit to parent young children. But he said infants "must want mums, no matter how you look at it".
"I think it is a bit strange if they choose dads," he said.
His remarks reflect Japanese reality. According to the BBC, Japanese fathers spend less time doing housework and taking care of their children than men in much of the world.
In the United States, men spend about three hours a day helping with kids and chores. In Japan, it's one hour, according to the BBC. The average Japanese father spends just 15 minutes a day with his children. Just 2 per cent of Japanese men take the paternity leave they are entitled to.
Hagiuda also acknowledged that raising children is hard work and that Japan needs to "create a system to care for mothers".
His remarks came on the heels of several other sexist comments from Japanese lawmakers. LDP's Kanji Kato, 72, said this month that newlyweds should have at least three children. The remark prompted one female LDP lawmaker to accuse Kato of "sexual harassment". Yukio Edano, who heads a different Japanese political party, called the statements "intolerable".
Hagiuda's remarks also sparked controversy.
Sumire Hamada of the Asia-Japan Women's Resource Centre called them "rude".
"What happened to the Government's pledge to build a society where men can participate in child-rearing?" she asked, according to the Japan Times. "These comments overturn what the government has said, and I'm sure many fathers have been angered."
Tetsuya Ando, founder of the Fathering Japan organisation and father of three, took issue, too.
"When he said children under three like mothers more than fathers, that's unacceptable," Ando told the Japan Times. "That kind of remark puts pressure on working mothers to stay at home while removing fathers' rights to rear children."
Japan has often struggled to incorporate mothers into the workplace. Things are so bad that in 2013, the BBC asked whether Japan was "the worst developed country for working mothers".
Women say the long hours, part of Japanese culture, make raising a child nearly impossible. Studies by the government show this is the main reason young mothers leave their jobs. "If you want to keep working you have to forget about your children, you have to just devote yourself to the company," Nobuko Ito, a lawyer, told the BBC.
Before she had kids, she would sometimes work from 9am to 3am. She left her corporate law job after giving birth. "I can't do this, it's impossible," she said.
Even women who want to work face obstacles. In parts of Japan, there is also a severe child-care shortage. The Tokyo government says about 20,000 children are on waiting lists for childcare spots.
The lack of women in the workplace is a big problem for Japan. The country's population is shrinking, and needs qualified workers. Prime Minister Shinzo Abe has made increasing the number of women in the workplace a top priority.
