Immigration Minister Iain Lees-Galloway said the treatment of the alleged victim was "appalling".
"I have absolute sympathy for her. This incident fell well short of the expectations I have, and I've been informed by Immigration NZ that they are taking action to ensure this doesn't happen again."
An internal review of the case is under way but changes have already been made.
A senior Immigration supervisor now has to approve decisions to detain someone in custody, which are then reviewed again after 24 hours and before the person is deported.
Staff who work with illegal migrants will also be given training on how to deal with victims of crime.
No decision has been made on whether criminal charges will be laid after her complaint, but police are still investigating the allegations of a serious crime.
Detective Inspector Colin Higson said the woman was provided with counselling and support after her complaint.
However, he said police were expected to tell Immigration NZ of people unlawfully in the country.
Higson declined to comment on the decision to deport the woman.
"The steps taken during an investigation are always on a case-by-case basis, and we work with partner agencies when required on what are often complex cases," Higson said.
"Any questions about the deportation of individuals from New Zealand should be directed to Immigration New Zealand."
Pete Devoy, the assistant general manager of Immigration NZ, said the organisation apologised "unreservedly" to the young woman and took full responsibility for the handling of her case.
"In the vast majority of cases our staff make the right decisions but in this case we accept that we got it wrong.
"As soon as senior managers became aware of the case INZ initiated steps to ensure the individual could return to New Zealand as soon as possible."
Devoy said police asked about the immigration status of the woman, then asked for an update one month later.
That day, she was taken into custody under the Immigration Act for four days - then deported.
As in every deportation case, Devoy said the young woman was asked in an interview to state any "humanitarian circumstances" she wanted taken into consideration.
The decision to deport was approved by the immigration officer's manager.
However, Devoy said the manager was unaware of the criminal complaint laid by the woman and therefore her "status as a victim and witness".
"Immigration New Zealand accepts that it was inappropriate for her to be deported given the pending criminal complaint made by her and the related police investigation."
Her family laid a complaint with their local MP a week after she was deported, and Immigration NZ became aware the next day.
"Immigration New Zealand has been in regular contact with the family throughout this process and has also met them to apologise for the distress caused and discuss the broader issues associated with the case," Devoy said.
