Ensuring they were properly rehabilitated would make it easier for ex-prisoners to integrate back into society and reduce reoffending.
He also revealed he would review how sentencing and bail was being managed by the courts.
Part of the reason our prison population was so high was because we were jailing people for longer, he said.
Little said the issue was not necessarily with the legislation, but instead more likely stemmed from how it was being applied and enforced.
"What we do have to do is get some consistency."
Although judges were doing a good job and interpreting the law the best they could, stronger guidelines were needed.
One way to do this would be by establishing a sentencing council, he said.
Little would also look into how bail was being administered. He questioned whether it was reasonable to lock up many people who had been charged but were yet to be tried.
While there was "no question" the safety of the community needed to come first, it needed to be balanced against the actual risk offenders awaiting trial posed to the public.
He said it was "thoroughly enjoying" his role as Minister of Justice, Minister for Courts, Minister for Treaty of Waitangi Negotiations and the Minister responsible for the GCSB, NZSIS and Pike River re-entry.
"It's been a fascinating year."
However, he had also "thoroughly enjoyed every minute of being Leader of the Opposition".