Mr Key stood by his decision to attend Chogm despite calls for him to boycott it because it provided the chance to ask about issues such as political disappearances.
"If we hadn't come to Chogm, we wouldn't have had that conversation. He was very animated, and aware of the issues. We didn't sugarcoat them. We specifically said to him 'our cameraman has been to the area, filming these people and what is your response to that? He very directly came back and disputed it."
He said he could not verify President Rajapaksa's assurance that there was a robust process for investigating the whereabouts of those who were missing, but nor could he refute it. The President had also told him some of those unaccounted for were boat people who had gone to Australia.
Mr Key remained concerned about the high military presence still in the north.
"But what he is very conscious of is that everyone is looking at that and monitoring it. It is very much on the international radar. Long term, for Sri Lanka to be operating as a country should, there is no question they have to have full reconciliation. These things take a long period of time."
Mr Key said the President's main argument was that it would take time to resolve the issues needed.
Mr Key said it was realistic to acknowledge the process of addressing such issues and reconciling with the Tamil population would take some time given it had been a long and bloody civil war which ended only four years ago. Mr Key has compared that to the Treaty of Waitangi settlements which came decades after the Maori Land Wars.
"I said to him almost every country goes through this process but what is important is that you do get through an effective form of reconciliation. You do have to hear people's stories and you do have to do everything you can within the bounds of reason to find a pathway forward where there is a place for everyone. Even in New Zealand you can see how long that has taken."
He said New Zealand would not go soft on such issues for tradeoffs in other areas, such as the promise to help send elephants to Auckland Zoo.
"We are comfortable that they come to New Zealand, but our relationship with Sri Lanka is one that is developing. The fact they send a couple of elephants isn't going to stop us raising hard questions in a meeting with the President."
He said the elephants for Auckland Zoo was a matter between Auckland Zoo and the Sri Lankan Government which had been in talks over it for the past few years. That was being led by Sir Don McKinnon on behalf of the zoo.
Labour's foreign affairs spokesman, David Shearer, said Mr Key had damaged New Zealand's reputation as an independent and honest broker and strong defender of human rights.
He said that while British PM David Cameron had taken the step of travelling north to speak to the families affected by displacement and alleged war crimes, Mr Key had done little and would not even call for a proper investigation.
Mr Cameron left after demanding moves towards an independent inquiry into alleged war crimes in 2009 by next March, when the UN Human Rights Council will meet.
Mr Key said while things in Sri Lanka are "not perfect'', the country had made progress.
"They are certainly working on the issues of reconciliation and devolving power now to the new Tamil Government that's in place really in the northern part of the country,'' he told Newstalk ZB.
The station questioned Mr Key on how he and British Prime Minister David Cameron could come away from CHOGM with seemingly opposite positions. Mr Cameron called for Sri Lanka's President Mahinda Rajapaksa to hold an independent inquiry into his country's alleged human rights abuses, or face a UN investigation.
However, Mr Key said he didn't think the two views were "entirely different''.
"I think it's important to understand there's been two [inquiries] - one that the Government themselves have done, one that the UN did,'' he said.
"Our main point, and in fact, what David Cameron is saying, he's saying he expects more to be done on the, not so much on the reconciliation - although that's one part of it - but on the accountability issue, and he's saying that if that doesn't happen then he might raise it with the UN Human Rights Council in Geneva in March.
"Well ok, fair enough. And New Zealand might support him on it. Let's see how that all goes.
"All I'm saying is that as much as accountability is important, probably the next step is really one of reconciliation and trying to find a way where peace can be sustainable in this country.''
- additional reporting, APNZ
