It says the influx of thousands of athletes and spectators into Brazil will accelerate the march of the virus.
In a damning assessment of the relationship between the WHO and the IOC, the letter questions whether the UN health agency is able to give a non-biased view of the situation.
The letter states: "We are concerned that WHO is rejecting these alternatives because of a conflict of interest. Specifically, WHO entered into an official partnership with the IOC, in a Memorandum of Understanding that remains secret."
The letter calls on the UN health agency to disclose the memo. "Not doing so casts doubt on WHO's neutrality," it said.
"WHO must revisit the question of Zika and postponing and/or moving the Games. We recommend that WHO convene an independent group to advise it and the IOC in a transparent, evidence-based process in which science, public health, and the spirit of sport come first," the letter said.
"Given the public health and ethical consequences, not doing so is irresponsible."
One of the letter's signatories, Professor Edwin van Teijlingen, an expert in reproductive health at Bournemouth University, accused the IOC and the WHO of being "too close for comfort".
He said: "There is a Memo of Understanding between them which they have not made public. They are not democratically elected organisations."
In Brazil, nearly 1300 babies have been born with the devastating Zika-linked syndrome known as microcephaly, in which babies are born with unusually small heads and brains.
However, Nyka Alexander, communication officer of the WHO, said: "Stopping the Games would not stop Zika. Because of how much humans travel around the globe and because of how much mosquitoes travel, Zika is going to make its way to other countries."
She added that there is currently no Memo of Understanding between the IOC and the WHO, and "certainly none involving any financial agreement or relationship".