Mr Baty told the Herald today if the other three people aboard Mr Baty's boat took polygraph tests and passed, he would pay the prize immediately.
But one of the three, Lance Young, said the experienced fishermen would not take the test.
"I don't believe in it. [The failure] was probably just a nervous thing. How can you fail? We saw it."
Instead, the men were relying on witnesses who saw what happened and were prepared to put their name to affidavits, along with GPS coordinates from their electronic fish-finder and Go Pro footage of the fish being reeled in.
One of the eyewitnesses, Tony Baxter, said he was fishing around Mr Young's boat all day, about 100m from Cape Kidnappers.
The boats were well within the tournament boundaries when Mr Young hooked up, he said.
Carl Ferris said he was also nearby, saw Mr Young had hooked something and was battling with it, and kept at a distance.
"We were there. We were pissed off we didn't catch it."
John Payne said he was trolling for marlin in the same area and watched as Mr Young reeled in his catch.
On Newstalk ZB, Mr Young said he hadn't heard anything from Mr Baty since he went public about the matter last week. He was unsure how he failed the polygraph.
"I've told the truth right through the interview and everything I've done."
Mr Baty said he had not turned down the claim but wanted more information, including technical data.
He said his company had written 23 contracts with the club, dating back to 2002.
The club had a "high claim rate" and after a $40,000 prize was paid out, Mr Baty said a polygraph would be conducted over the next claim.
"We've had a 15-year relationship with this club ... every single contract has had a polygraph clause so this is not eye-opening, just happened," he told Newstalk ZB.
"I am trying to conduct an investigation. What we are trying to get out of the club is them to ask these guys to get the technical data on the boat."
The man who conducted Mr Young's polygraph test stands by the results, but admits the tests aren't perfect.
Craig Gubbins, from Personal Verification, said he had no qualms about the reliability of the polygraph test. The results were highly conclusive, and there was no doubt about the results from the 33 questions he asked Mr Young.
He said there was an error rate in every test and validity ranged from 89 to 94 per cent.
Mark Handler, from the American Polygraph Association, said it was customary to review test results and he had approached the fishing club to do so.
Fishing club president Alex Smith has said the club agreed to the contract with OddsOn Promotion that included the polygraph test clause. It backed Mr Young and the other three men.
The club had agreed to help Mr Baty investigate but thought requests to send the boat's electronics to their manufacturer were a step too far.
