Simon Bridges says Jami-Lee Ross would have been kicked out of the National Party much sooner if he had known about claims of sexual relationships and harassment that emerged today.
National's leader said he was "gutted" to read a report that two women had sexual relationships with Ross, while two others said he had harassed them.
"I'm gutted about the story and everything it's about," Bridges told reporters.
"I am in admiration of the courage of these women."
Bridges said he did not know the women's identities.
If he had been aware of inappropriate behaviour of Ross, he would have kicked him out of the party "much earlier".
But as soon as he was aware of inappropriate conduct claims, "I acted immediately."
Bridges said Ross was a "deceitful person".
National's leader also insisted he retained his MPs' support: "55 MPs are unified behind me."
Ross 'not bloody bringing us down'
Meanwhile, Paula Bennett has declared Ross is "not bloody bringing us down" - and described the women who had personal relationships with the rogue MP as "victims".
National's deputy leader described the women who came forward today to allege being "manipulated and intimidated" by Ross were "incredibly brave" for speaking out against Ross, whose departure from the party has seen a week of extraordinary political carnage.
"We have had appropriate supports around anyone who has come to my attention. I have always put the women at the centre of everything I have done," Bennett said.
She had only become aware of the allegations against Ross ahead of a meeting at which he took leave from Parliament at the beginning of this month.
"When matters have been brought to our attention, we have dealt with them immediately. When inappropriate behaviour was raised with us, we took this to him."
She said she was unaware what was meant by comments from one woman which suggested the National Party had earlier been forced to intervene over Ross' alleged behaviour.
"We will regroup and make sure all of our people are safe."
Bennett would not be drawn on the identities of the women or the roles they held. She would not comment on whether there were sitting MPs or media among those considered to have had relationships with Ross.
"In situations like this, often other things come to light."
Bennett said she was unsure what would transpire next in a week of revelations during which Ross had drip-fed accusations, a covert recording and had promised to release private text messages.
"I choose not to try and go into the mind of Jami-Lee Ross. That seems like a very dark place.
"I don't care about stupid things people may or may not say (that have been recorded).
"I care about the fact there are women who feel victimised and hurt and not safe in that situation. I think that trumps everything else.
"We've got a great team. He's not bloody bringing us down."
Ross reportedly had relationships with four women, with some claiming they were groomed and used for access to information and power.
One of the women, who worked in the same political circles as the married MP, says she had an affair with him after constant and repeated pressure, Newsroom reports.
Two of the relationships were sexual, the website says. Some of the women, who are not identified, felt "manipulated and intimidated" by Ross, while others felt pressured not to speak out.
One of the women, who said she had a sexual relationship with the MP, said Ross had told her his marriage was "on the rocks" and promised their relations would remain secret.
"Initially it was euphoric but I knew it was wrong," she told Newsroom, describing Ross as a "narcissist".
"It was like a fairy tale. I'd found this person, this soulmate, who provided meaningful advice … he always knew what to do. Looking back, it was nothing but a game."
But things changed after Ross began asking where she had been and who she had been with, she said.
"It was very clear to me his political motivation was a lust for power and control," Newsroom reported.
"When I left, I saw him for what he was and only then could I fully protect myself. That was when a wave of people started coming to me to say 'Me too'."
A second woman said she had an affair with Ross, another claimed she was abused by Ross and another who said her interactions with Ross has "destroyed" her as a person.
Ross, who has been expelled from National after a review concluded he leaked Bridges' travel expenses, told media yesterday that he was happy with how he conducted his private life. He warned against National trying to "lift the bedsheets".
Following his two-hour meeting with police yesterday about claims of electoral fraud, Ross addressed allegations of harassment.
"I had harassment allegations that supposedly existed around four women. I'm very clear on my recollection. I have further recordings of Simon Bridges and I in his office. I'm confident in what I said yesterday.
"I know that there's smears about me at the moment: what I think has always been something in New Zealand politics that we leave personal lives and family out of this.
"I'm comfortable with what I have with my wife - if the way in which we're about to play politics, when a political party and the leader is under pressure, if they want to start lifting the bedsheets on everyone that works in that building, you guys and MPs, then I think there'll be a lot of people concerned - even those that are throwing allegations now."