Armstrong confessed to being a cheat during an exclusive interview with Oprah Winfrey on Tuesday and has been accused by one anti-doping agency of masterminding "the most sophisticated, professionalised and successful doping programme that sport has ever seen".
Mr Avery said that as a result, Armstrong had greatly damaged the sport.
"It's good they caught him and it shows that even after you retire you can still be caught.
"However, it has already damaged the sport. Everyone I meet that asks what my profession is, when I tell them, they look at me in disgust and the first thing they reply is 'so do you take all that **** too'?"
Mr Avery said when he replied "no" people would often look at him as if he were lying.
"His [Armstrong's] confession is just going to make these situations even more awkward."
He said the good news was that since Armstrong's glory days the sport had come a long way in redeeming itself.
He said the recent introduction of a blood passport meant pro cyclists must complete testing before racing.
"Any value change in that blood passport results in you being investigated," he said.
Rotorua's Bryce Shapley, who raced professionally in Europe during the turn of the century, said he did not feel sympathy for Armstrong.
"I'm sympathetic to those people who knew he was guilty and fought against it so hard and I feel for those riders who were robbed of placings."
He said he hoped Armstrong would give back to the sport by working to help stomp out drugs in the sport during the future.
Despite the damage done by Armstrong, Mr Shapley said the sport had a bright future for clean riders.
"I certainly believe the sport has come a long way."
He said just looking at the number of New Zealand riders racing in Europe now, like Avery and Sam Bewley, showed clean riders were getting more of a chance to compete.
Mr Shapley said he had experienced racing against riders who were using performance enhancing drugs, and it ruined the contest.
He said even at the height of his fitness in Europe he would often feel he "was going backwards" when compared with some of the other riders.
"I couldn't understand how some of those guys could go so fast, I'd wonder what was I doing wrong?"
Mr Shapley said looking back it was obvious drugs had a big part to play.
The highly-anticipated interview between Oprah Winfrey and Lance Armstrong, which includes his confession, airs on Friday at 3pm on the Discovery Channel.