The victim, who is over 16 years of age, has name suppression.
Judge Hobbs said Kay was not to have contact with the victim unless it was to do with the restorative justice process.
Kay has been relieved of his duties at St Mary's, the parish he joined in January 2013.
A spokesperson for the New Zealand Catholic Bishops would not confirm whether the victim had an affiliation with St Mary's Church.
The spokesperson also said everything was being done to co-operate with the police process, and to support the parish.
Monsignor Gerard Burns, a visiting priest, has arrived to support the St Mary's Parish.
