Most of the money people put into KiwiSaver they would have saved anyway.
Most of the people KiwiSaver was especially intended to benefit are not in the scheme.
Most of the scheme's taxpayer subsidy leaks to those outside the target group.
And while KiwiSaver's effect on national savings is positive in the short term, on a longer-term view it may be negligible or even negative.
These are the takeaway messages from work Treasury economists have done crunching the numbers from a survey the Government commissioned Colmar Brunton to undertake last year as part of an ongoing process of evaluating KiwiSaver's effectiveness.
The results were presented to a recent New Zealand Association of Economists conference in a paper by Grant Scobie, David Law and Lisa Meehan.
They scrupulously avoided drawing policy conclusions but similar points run through the Budget advice the Treasury released last month.
And it helps to illuminate one of the puzzles of this year's Budget: why in an election year would a Government hack into a scheme to which three out of four working New Zealanders belong?
The Budget changes reduce the attractiveness of the scheme but also its fiscal cost. The member's tax credit is halved and the minimum contribution raised from 2 to 3 per cent of pre-tax income.
The employer's subsidy will be raised to 3 per cent too, but by eliminating the tax exemption on those payments, the new policy ensures that either most or all of the increase will go to the Government, depending on the member's marginal tax rate, and not into his or her savings.
The Colmar Brunton survey, which predated those changes, involved face-to-face interviews with 825 people aged between 18 and 65.
It comes with some health warnings.
It was conducted when KiwiSaver had been going for less than three years, so its conclusions would not necessarily hold good when the scheme has matured.
And as a snapshot at one point in time it cannot show how particular individuals' behaviour has changed over time. For that we will have to wait for the next crop of SoFIE data (Statistics New Zealand's longitudinal Survey of Family Income and Employment) next year.
That said, its results are revealing.
On average respondents who were KiwiSaver members said they would have used 64 per cent of the money they were now putting into Kiwisaver for other forms of saving or for debt reduction.
That implies that only 36 per cent of the member's contributions to KiwiSaver constitute additional saving.
But the economists reckon the "additionality" is less than that, on the grounds that people on higher incomes would have been saving a higher proportion of their incomes and therefore had more scope to switch to KiwiSaver.
They estimate that each additional dollar a member puts into his or her KiwiSaver account represents on average a net increase in savings of just 29c.
One of the Budget advice documents makes a similar point, citing a Ministry of Economic Development study: "While total KiwiSaver assets grew by over $4.7 billion between March 2008 and March 2010, the total of all managed funds, including KiwiSaver, grew by only $2.5 billion, indicating some degree of switching away from other savings products to KiwiSaver."
Next the Treasury economists looked at the issues of target effectiveness (how many of the people the scheme was particularly designed to benefit belong to it) and leakage (how many of those benefiting from membership fall outside the group the scheme was targeted at).
The act establishing KiwiSaver says its purpose is "to encourage a long-term savings habit and asset accumulation by individuals who are not in a position to enjoy standards of living in retirement similar to those in pre-retirement".
The survey asked two somewhat different questions from that. Did respondents expect their income in retirement to meet their basic needs? And did they expect it to reach the levels required to be comfortable?
It only looked at a subset of respondents, those 25 or older, who had done some financial planning and could provide estimates of their expected retirement income.
And what would count as basic needs or as a comfortable standard of living was left for the respondents to judge.
On that selective and subjective basis, the survey found that only 35 per cent of those expecting their retirement income to fall short of meeting their basic needs were in KiwiSaver, and only 46 per cent based on being comfortable.
Conversely, 80 per cent of those in the scheme did not need to be in order to meet their basic needs, and 43 per cent did not in order to be comfortable.
It gets worse.
For KiwiSaver to be effective it is not enough that people with an expected income shortfall join the scheme. They must also reduce their consumption spending and increase their retirement saving over and above what they would have done if they had not joined KiwiSaver.
On that basis the Treasury economists estimate that 93 per cent of the benefit leaks, on a basic needs criterion, and 78 per cent on a comfort criterion.
"The [fiscal] cost per member from the target population based on basic needs is over $13,000 a year and based on being comfortable it is around $4000 a year."
When it comes to KiwiSaver's effect on national savings, the paper acknowledges that it boosts them in the short term.
The Government subsidy is a wash from a national savings point of view, if you make the debatable assumption that every dollar the Crown contributes to someone's KiwiSaver account is a dollar it has to borrow.
But even 29c in the dollar of additional saving by the member is additional saving, and there is the employer subsidy as well.
In the longer run, however, the risk is that as KiwiSaver balances build up, people will look at every extra dollar going into their account, regardless of where it came from, as one less dollar they need to save in order to reach some target level of savings.
"The net contribution to overall saving would be marginal at best in the longer term and may in fact reduce national savings."
Whether people actually have in mind some target level of retirement savings is debatable, however.
And the Treasury's Budget advice, when talking about modelling the effects of changes to KiwiSaver on national savings, warned that such estimates necessarily depend on assumptions about how individuals and businesses would respond to changes in incentives and regulatory requirements. "Hard evidence on these behavioural responses is limited."
Undaunted, it estimates the Budget changes to the scheme, designed to transfer costs from the public to the private sector, would reduce the country's external debt by around 2 per cent over the next 10 years compared with what it would otherwise be.
Not trivial, but not the difference between safety and peril either.
READ THE FULL REPORT