"It appears Smith and I have two things in common; we both had eyes on the same female and we are now in the dogbox for it," he said.
As he made a formal apology to the Otago halfback and rugby's governing body the man said the man revealed there had been considerable backlash since the story broke.
"We have had a lot of abuse since the story broke," he wrote.
"It was actually my wife who wished to go to the media and it has therefore caused a lot of our strain in our relationship this past week, mainly due to the reported comments that I found the lady attractive."
He said because of the incident blowing up more than expected they did not intend distributing the video of the toilet tryst.
At the time the man said he did not have a problem with Smith's actions, finding them "gold" and "quite hilarious".
However, his wife said Smith's actions were disgusting and having sex in a public toilet in an All Black's uniform was wrong.
Smith was slapped with a one-match ban after the episode, and returned from South Africa at the weekend.
He now faces the prospect of further disciplinary action following a New Zealand Rugby investigation.
Before his flight he gave a tearful apology in front of media; apologising to his fans, family and his partner, Teagan Voykovich.