The most recent new fee is a new insurance charge that will be borne by all port users from October 1, as earthquake damage to ports in Wellington and Christchurch means all New Zealand ports are considered high risk.
Port CEO Garth Cowie said this meant insurance premiums had more than doubled "with less than seven days' notice".
"The increase alone was more than $2 million. Our deductible has been doubled from $12.5m to $25m. We have been absorbing these costs since July 1 in a bid to give our customers as much notice as possible."
The port said it had to pass those costs on by introducing a range of levies. It has reduced these fees in light of concerns.
However Mr Taylor and other port users still face a charge, which they would have to pass on to their customers.
"Although many of our customers will agree to pay it, some will refuse and force us to call their bluff and they'll probably go out to the market," he said.
Importers and exporters could choose to use another port, "which leaves us as a business high and dry."
Port users are reportedly considering taking legal action with growing concern the publicly-owned port is out of touch with its users.
There has also been talk of refusing to pay the fees, as floated by Lowe Corporation's Andy Lowe.
"The decisions being made by Napier Port staff and the Port Board are scarcely believable. They need to be held to account by both customers and their employers, the ratepayers' of Hawke's Bay," he said.
He added the port's owners - the Hawke's Bay Regional Council - should be asking the port to justify increasing charges.
Council chair and former grower Rex Graham said he had discussed the charges and the port said they were legitimate and justified.
Political consultant Simon Lusk said the port had "committed the cardinal sin in Hawke's Bay politics of angering the apple growers"
"The $100 TEU fee during the peak season is a call to arms to the most militant group in Hawke's Bay. This will be a massive cost to the industry so they will be willing to invest heavily in fighting the charge," he said.
"The port should immediately sue for peace or the apple growers will do to port staff what they did to former Hawke's Bay Regional Council CE Andrew Newman. You simply do not pick a fight with the apple growers if you have any sense at all".
Members of Hawke's Bay industry have met with the port, which resulted in the port reducing some of the charges and adapting the application of the levy.
Mr Cowie said they understood the impact price changes might have on their customers, and had worked with them to "find some middle ground."
There has been a call for the insurance costs to be passed onto the port's main customers - the international shipping lines - as it would have less impact than for a local business.
Mr Cowie said they would be reassessing this levy after 18 months.
By this time the port would have renegotiated insurance contracts and could take any reduction into account. Global shipping forecasts showed it could be better to charge the levy via shipping lines at that time.
Mr Cowie said the port had also sought feedback from the pipfruit sector on the concept of a peak season reefer surcharge.
"The apple industry is growing and Napier Port's infrastructure has to keep pace in order to support our growers and provide the level of service they need," he said, adding this came at a cost.
Over the past five years the port had invested more than $95m in infrastructure.
"We have done everything we can to keep this proposed fee to a minimum while still ensuring the pipfruit industry has the infrastructure they need for the peak export season."
The number of apple exports through the port has increased from about 12,936 containers in 2008 to a record 22,205 20ft containers of apples last year.
