"I ended up caddying for him by default and took it on because I knew he wouldn't sack me. That's how we started.
"He is a free spirit and so positive mentally and never expects to hit a bad shot. I used to ask him what he was going to do and he would say; 'Geoff, I will show you a magic shot.'"
Smart's regular response was to tell Lee he was there to watch not to tell him how to play.
In many ways, Smart felt the 25-year-old Lee was swamped by too much advice from a multitude of coaches when he took to the PGA Tour.
It took him time to emerge from advisers who were telling him how he should play the game.
"Danny does not see the bad shots, he is totally amazed when he hits one," said Smart.
"I think he is just now playing his own game and feeling comfortable or maybe he has adapted to a slightly more conservative way I'm not sure. But I think he definitely would have had conflict about being told how to play.
"I'm sure that was an issue for a while when he went through lots of caddies and lots of coaches over in the States."
Lee won his first PGA tournament in July, the Greenbrier Classic, made his first cut in a major and finished in a tie for 43rd in the USPGA and has earned US$3,233,397 ($4,882,061) this season.
He's had seven top 10s, a scoring average of 70.443 and is the busiest golfer amongst the leading money-winners with more than 100 rounds of tournament golf this year.
Lee is playing his own game, one he honed on the practice range and Springfield course in Rotorua and in regular visits to work and learn from Steve Jessup an hour away at Wairakei.
What is that game, how does he compare with others?
"He backs himself to have an aggressive game," Smart said. "He is more confident now that he is achieving results to go alongside his style.
"He never had any trouble getting birdies and he's probably having the same number now and less big numbers on the card. You have to have lots of birdies to be able to win on the PGA tour."
When Lee turned up at Springfield, he got enormous help from former national and BoP rep Mark Smith who got to know the youngster's parents. Smith captained the side and he and Lee would always room together at tournaments and on rep duty.
"He was only a young fellow, hadn't been in New Zealand for too many years and didn't know too many other people. He adapted pretty quickly to most things though and we'd see a lot of each other on the course and outside.
"We used to give him a bit of stick because his English wasn't too flash then.
"It's not as bad as it might sound [Lee talks in a slow, staccato manner], he talks slow in Korean as well," Smith said. "That's how he speaks and he has a good sense of humour.
"He speaks and understands English well but it's how we hear him talking a bit like a machine."
There were times, Smith said, when he had to growl at Lee for saying something inappropriate but that was only because the youngster did not always grasp the impact of his adopted language or the context of the situation.
At one interprovincial competition in Ohope, Lee's opponent snap-hooked his opening tee shot into trouble and Lee, in his slow, inimitable voice, said: "That's one down."
Smith growled at him while Lee, who was only about 14, was amazed because he had simply uttered a fact.
The youngster didn't get the same sort of protection when he played Texas Holdem on tour with his provincial teammates. He was equally aggressive at cards and often fell to his teammates' wiles.
"He doesn't know any other way at golf, I'm not sure why that is," Smith said.
It might have been something to do with learning to play at Springfield, which was a short course and forced everyone to learn all the aspects of the game.
It was tight with several dogleg holes, plenty of trees and you had to learn to hit the ball high, low and bend it both ways.
Several par fours were driveable and the par fives could be reached in two if you hit a strong tee shot.
"That design teaches you to be aggressive but you have to hit it well, you can't spray it at all."
Lee was "freaky good" with his wedges from 120m in and his putting was not sharp but not bad either. He didn't need to putt too well because he hit it so close to the flags.
Most days he would practise for four hours and play a round while on a rare day off he would still practise for four hours. Unlike a lot of good amateurs Lee could go from the range and then deliver on the course.
He was a regular visitor to Taupo where he was coached by Jessup for a few years until he turned professional.
"I'm not too sure about the extent of his school work but Danny would come across to do 40 hours a week at Wairakei," Jessup recalled.
"Even then as a 15- or 16-year-old he would always wear longs because that's what professionals did. His parents would drop him off, go back and pick him up at night. He spent more time here than some of our staff.
"I know, in his amateur career they were trying to get him to manage his schedule a bit more but I don't think that suited him. One of the reasons he's going so well now is that he is playing nearly every week."
Lee first approached Jessup to inquire about his putting and then they moved on to his swing.
Putting was and still is the weakest part of Lee's game and he still looked a little confused about that discipline. He began with a long putter and was always searching for an answer.
Lee was never totally comfortable with his swing and most of his work with Jessup was to try to quieten down his hands. Conversely, his wedge play was terrific.
At one Australian Open, caddy Michael "Sponge" Waite, who had caddied for Michael Campbell, was blown away when he watched Lee's wedge game.
At a visit to the Masters in 2009, club manufacturer Roger Cleveland spent an afternoon with Lee and Jessup and remarked that Lee's wedge play was as good as he had ever seen.
"He is the guru on designing wedges and most of that brilliance was on Danny's natural ability, we didn't do a lot of coaching on that.
"Danny is just so determined and everything you would want from an athlete."
