But Oliver does not see it in those adversarial terms.
"We are taking money from people [that] they would prefer not to pay," he said. "But we have built up quite a high degree of co-operation, within some constraints."
He attributes that to the fundamental coherence of the tax system following Sir Roger Douglas's radical reforms in the 1980s.
It has an internal consistency, compared with many tax regimes overseas which are much more of an accumulation of ad hoc concessions to special pleading.
"We have a lot of buy-in from the private sector. They understand it."
This was evident in the recent review undertaken by the tax working group chaired by Victoria University's Professor Bob Buckle.
"It means that in New Zealand when we are talking about tax reform it is a rational debate, not a lolly scramble.
"There is an expectation by everyone that you will have rational arguments for reductions in tax, which is not the case, in my experience, in countries where the private sector just wants anything that reduces their tax."
The high-water mark of coherence in the system was when the company, trustee and top personal tax rates were all aligned, greatly reducing both the incentives and the opportunities to dodge tax.
"But I think there is a natural instability in coherent tax systems. They are very constraining on governments. They don't allow them to easily alter tax rates," he said.
"There is a natural tendency, understandable given political pressures, to play with the tax system. They may want to demonstrate that high-income earners are paying a lot of tax, but at the same time they see that causes some economic damage in terms of investment and work incentives and so on, so you end up with a lot of loopholes."
Another source of pressure is globalisation and the mobility of capital.
While this inevitably puts downward pressure on the company tax rate, Oliver rejects the idea that capital will inevitably flow to wherever the tax rates are lowest.
"That's because tax pays for things that businesspeople value - law and order, property rights, infrastructure, medical care and schools and so on."
Ultimately, however, we may face some hard choices, between maintaining the internal coherence of the existing system and its international competitiveness.
A rethink, perhaps along Nordic lines with very different tax rates for labour and capital income, might be needed.
But for now "a little bit of incoherence you can live with".
It was a mistake, Oliver acknowledges, to persevere with an international tax regime that tried to tax all of New Zealanders' global income.
"Theoretically it was the right approach but as Michael Cullen said we boldly led where no one followed. We probably persisted with it too long and it it probably did stymie the development of New Zealand companies offshore."
It never made intuitive sense to taxpayers and it is important tax policy do that, he said.
"Take, for example, GST and its single rate."
Economists might argue for different rates depending on the elasticity of demand for different goods and services. Things which are least sensitive to price, such as water or medical services, should be taxed harder, and luxury goods most lightly. In theory.
"But frankly that would make no sense to anybody. People would just think that is nutty. They would think it is the opposite, that water should be taxed lightly or not at all and luxury items highly taxed, that that is fairer," he said.
"But everyone can understand a single rate."
The tax system should have modest objectives, Oliver said.
"The purpose of taxation is fundamentally to raise revenue in a reasonably fair way. To use it for other things is extremely dangerous because there are so many unintended results."
Oliver's opposition to a capital gains tax is pragmatic rather than dogmatic. "IRD did not see it as worth the candle. Treasury did," he said.
"You would have to exempt the family home and it would have to be levied on a realisation [rather than accrual] basis, which creates a lot of complications and means there would be little immediate revenue in it."
Nor is he persuaded by the argument the OECD's tax economists run, that it is better to tax land, which can't leave the country, than capital or labour which can.
"The problem is it is a windfall loss to existing landowners. The price of land would go down to compensate for the tax, so it doesn't hit new owners, only existing ones, who tend to be the elderly, farmers and Maori," he said.
"Is that fair? Why pick on them?"
The generic tax policy process provides multiple opportunities for consultation and feedback from tax practitioners and other affected parties before tax law is enacted.
It works superbly, Oliver said.
It is not just the intimate democracy effect of being a small country with short lines of communication.
"It is also that tax debates occur within a framework that everyone understands and that has massive buy-in from the private sector, so that it is not just a dog-eat-dog scramble."
He also credits the politicians, both ministers and members of the finance and expenditure select committee which examines tax legislation.
"On the committee the politicians are not just slinging things at each other. They get on with the job. It is a highly co-operative environment."
"There is a political element to tax policy, of course, but it tends to be about rates or whether to have a capital gains tax or alter GST.
"The vast bulk of tax bills go through unanimously."
Before joining the IRD Oliver had worked for Arthur Andersen, McLeod and Lojkine, and KPMG, as well as a stint at the Treasury.
After he leaves on November 7 he will be teaming up as a consultant with former Deloitte partner Mike Shaw.
His successor is Struan Little, previously a deputy secretary of the Treasury.
