Koya told Fiji Broadcasting Corporation he believed New Zealand and Australia had backtracked on commitments for a development-focused agreement.
He was concerned the labour mobility and development clauses of Pacer Plus were non-binding which meant they could withdraw at any point.
He also believed there was not enough protection for small businesses.
Papua New Guinea has also withdrawn from the agreement saying it did not believe it would directly benefit PNG but its Prime Minister Peter O'Neill has said it would benefit other countries and should be concluded.
Fiji had withdrawn from Pacer Plus negotiations during the coup years but re-joined in September 2014, saying it wanted "tangible and binding commitments" from New Zealand and Australia in the agreement.
A spokeswoman for McClay said the decision to pull out was not related to the New Zealand Government's concerns about the detention of Opposition MPs and a trade union official in Fiji over the weekend.
Foreign Minister Murray McCully and Prime Minister John Key have voiced concern after two Opposition party leaders and a trade official were taken into custody for taking part in a forum on the Fijian Constitution.
Fiji's law requires a permit for public gatherings. McCully said while the facts were still unclear he was taking a "close interest" and would be concerned if human rights or the freedom of political discussion were infringed.