"In that instance I felt like I knew what my skill sets were and I had an idea what I wanted to do after the show [HIMYM].
"I was surprised that [CBS] pitched me that idea, and I sort of sat for a minute with it and told him the things that concerned me about the longevity of that kind of gig.
"I think I would get bored of the repetition fast, and the structure of it is so set that I don't have any interest in doing monologue, commercial, sketch, sketch, guest, musical act, goodnight," he said.
Instead of taking over from Letterman when he retires in 2015, Harris pitched the idea of a weekly TV show.
"I like a weekly variety show, I think if it is weekly and you have some really great s*** on there, then you are going to get the guests to want to come," he explained.
When asked if the network was interested in that idea, he said: "[They] still might be".
Stern then suggested Harris might make a good replacement for Craig Ferguson's Late Late Show, which screens an hour later than Letterman's in the States, but Harris also shot down that idea.
"It's still nightly, you're still coming out with the same content," he said. "And now you're just getting bitter that no one is watching."
Listen to an excerpt from Harris' interview with Stern below:
- nzherald.co.nz