The man, who stood in the dock with tears welling in his eyes, leaned against a wall with his head down seemingly unable to look at those who had come to support him.
Some began crying as he was called to the dock, including an elderly woman who quietly sobbed through the entire court proceedings.
As the accused was remanded in custody to reappear on Thursday several raised their hands in the air to get the man's attention before saying "we love you bro".
The baby boy, a twin, who was admitted to Waikato Hospital with head injuries and was later moved to Starship Childrens' Hospital died overnight.
Acting Waikato District Crime Manager, Detective Inspector Karl Thornton, said "Our thoughts and condolences are with the family at present".
"We have a substantial number of enquiries to complete before we can establish exactly what has occurred."
Mr Thornton said a decision on whether further charges - if any -would be laid would be made at the conclusion of the forensics investigation and witness interviews.
A police forensics team was at the cordoned-off property that was still under police guard this morning.
A post mortem was to be conducted in Auckland this morning.