Judge John McDonald said he understood the children were taken from their parents by CYF and given to their grandfather and his wife but wanted a confirmation that was what happened.
Defence lawyer Arthur Fairley sought a confirmation in court from his client.
Child, Youth and Family regional director Tai Tokerau John Langley said no concerns were raised during checks for a caregiver assessment before the children were placed with their grandparents in 2002.
He said CYF's involvement ended after the grandparents obtained permanent orders for the children through the Family Court.
"CYF first became aware of the allegations of sexual abuse in 2012 when the police advised us of the concerns. As a result, we completed a joint investigation of the allegations with the police.
"Part of this includes assessing a safe care arrangement for any children while the investigation is completed."
The man's offending also related to him sexually abusing a cousin who was given a hiding and became an "outcast" of her whanau, Judge McDonald said.
That victim spoke directly to the man in court and said it took more than 40 years for her whanau to acknowledge his offending.
She was visibly shaking and her voice quivered as she spoke in te reo and in English. At one point, she left the court sobbing while the judge highlighted the nature of the man's offending.