Speaking on her This is Paris podcast, she explained: "I felt like it was a safe place because I'd been going on Letterman for so many years. He'd always have fun with me and joke around, but I thought he would keep his word on this and I was wrong."
Hilton revealed that she told Letterman she would not go on his show again and was later sent a crate of wine by the presenter as an apology.
She recalled: "After it ended, I just looked at him and said, 'I'm never coming on this show again, you crossed the line.'
"I didn't tell him off, because I'm not that kind of person but I got angry and he was apologising and sent a case of Chateau Lafite wine to our house."
Letterman made a public apology to Hilton when she returned to his show the following year.
He said: "I found out afterwards I had offended you. I felt horrible about it because I'm not here to make enemies, honestly.
"So I called you, and you took the call, which I thought was very nice of you, and now you're back and thank God, and I'm terribly sorry."