"I'm totally embarrassed by the whole situation. It has knocked me around, of course it has."
Mr Bertrand claimed he was under strain after his town of Gracemere, near Rockhampton, was threatened by bushfires - which then lead to his "horrible mistake".
A spokesman for Queensland's Department of Health said the family who were already grieving the baby were "distressed" after hearing about the horrific incident.
The spokesman confirmed the vehicle that was supposed to be transporting the baby was owned by a private funeral company, not the government, however the incident was now being investigated by the department.
Queensland Health said it would no longer use the company.
"This is an incredibly distressing incident," the statement read.
"This transport was not undertaken by Queensland Health but by an independent private funeral transporter.
"We have stopped use of the service. We are writing to the contractor to ask them why they should not be permanently removed from State Government work."
A spokeswoman for Queensland Police said the organisation was aware of the matter and confirmed the baby's death was not suspicious.
The investigation will now be led by Queensland Health however police are making inquiries.