A shoreline search was also carried out on the island.
Queenstown Lakes Mayor Jim Boult said earlier today that he had heard the man was "youngish".
"It's terrible when things like this happen in the district and everybody I'm sure is thinking about him and his family," Boult said.
Area response manager Senior Sergeant Allan Grindell said yesterday that the man was on Stevensons Island when he swam out to retrieve a drifting boat but then got into trouble.
Another person swam out to help him but that person wasn't able to locate him, he said.
Local vessels and the Coastguard searched for the man yesterday and a helicopter with paramedics on board was also assisting.
LandSAR volunteers were also helping look for the man yesterday.
Police extended their sympathies to the man's family.
The death will be referred to the Coroner.