The man, who would not be named, is HIV positive and regularly requires medical tests and treatment.
"I told my specialist nurse that she could, and would, and was capable of doing that. So the hospital were warned two-and-a-half years ago and did nothing about it."
He is now considering legal action against the health board.
"It could have been stopped then if someone listened to me and took me seriously," he said.
As recently as two months ago the man said he warned hospital staff again, after it was indicated to him his former wife knew more than she should.
"I rang the hospital saying my privacy had been breached. I was waiting for them to tell me [what they were doing about it] and it didn't happen."
The man said he was saddened for his former wife, who rang him several days ago to explain she looked at his files "out of concern".
She said she was in trouble at work but denied accessing other people's files.
"For me personally, it was in excess of 36 times over three years. It's quite stressful for me."
The district health board declined to comment on the man's claims he had raised the alarm.
Staff members concerned at what information the woman knew contacted management and a four-month investigation was launched.
During that time the woman was monitored while working at the same desk and computer she had used to access the private information.
District health board chief executive Phil Cammish said 30 of the 48 people have made contact to discuss their breach. Five sought additional information and have been shown their records, he said.
"Protecting the privacy of our patients is at the core of our work. To have that patient privacy breached by an individual who had no care about her actions, is unforgivable."
Mr Cammish said as soon as the district health board had sufficient proof of serious misconduct, the staff member was dismissed.
Another breach victim, who is also HIV positive and has other highly-sensitive medical matters, said he was disgusted and distraught his information was accessed without consent.
"Based on the serious nature of the files she was accessing she should have been put on leave or something. I don't see any reason why she should have been able to sit at her desk after she was the reason of suspicion," he said.
The man, who flats with the ex-husband, recalled his flatmate telling him after each of the occasions he had raised his concerns with the health board and it was "as if they have just dismissed him".
The flatmate described the letter he received as offensive.
"What's upsetting me is they wrote a letter that basically said "sorry about that, thank you".
"It's hurting me to have this happen and have the hospital say 'it was her, we could do nothing about it'."
A woman who's frail mother received a letter advising of the breach said she was outraged.
Her mother used to work at the hospital and there were sensitive family matters in her mother's files.
What upset her mother the most, the woman said, was not knowing who the woman was.
The district health board has refused to publicly name the woman.
Health Minister and Bay of Plenty MP Tony Ryall said he was advised the district health board was providing more detailed information for those affected people who contacted them.
"There are very clear rules around accessing patient records and there are systems to record who does access records. And abuse of these rules is taken seriously. This was a very serious breach of privacy that has been treated with the utmost importance."
