The man said his brother's partner and 9-year-old son saw him tread water for a short time, then he went under.
"They had only just left. We have a beautiful photo of him two minutes before with his life jacket on, it's just a horrible accident."
The man's brother said they came from a big family and there was a lot of support for his partner and son.
"The locals have been extraordinary, they have put his son in the house next door with kids to distract him and welcomed us into their homes. They have just accommodated our every need.
"The police have been amazing and the coastguard as well. We had 14 boats searching after it first happened. It's been amazing, it's just an amazing community."
He said his brother was adopted and Australian police had gone to support his biological mother.
"There's a couple of lessons here, make sure it's not just one person who knows how to operate the boat on board and make sure your life jacket is fit for use," the brother said.