Hours later, a search was launched after boaties heard a man shouting out for help in the water near the Chelsea refinery shortly after 1am today.
The police Eagle helicopter and police launch Deodar searched for the man around the refinery and West Harbour.
The launch crew found his body in the water near the refinery about 2.05am.
Inspector Willie Taylor of police northern communications said the man, aged 20-30, had been fishing and drinking with four others at West Harbour.
He decided to go for a swim dressed in a short-sleeved black T-shirt and black underpants.
"He'd obviously got into some difficulty on an outgoing tide and had drowned after a period of time," Mr Taylor said.
Police believed alcohol, inappropriate clothing and an outgoing tide were factors in his death.