According to an excerpt from the book, Letterman: The Last Giant of Late Night, obtained by People Magazine, the 69-year-old feared he'd lost his family.
"It was akin to having killed your family in a car crash. It was like that to me," he told the book's author, Jason Zinoman. "I was afraid my family was gone."
Letterman told the author he talked to his staff as a way to "cope with and avoid his personal life".
"I was looking for a refuge. Whether they knew it or not, [my staffers] were being used to support me," he said.
"I didn't want to go outside. Outside I was scared. Scared as I've ever been in my life. The show was endlessly helpful."
Letterman broke the news of his affair to his audience in the now-infamous opening of his October 1, 2009 show.
"I have a little story to tell you," he said, before detailing how he found a package from Halderman that contained a letter that read: "I know that you do some terrible, terrible, things."
Later in the show, Letterman confessed: "I have had sex with women who work for me."
Letterman's wife, Regina, stuck by him, as Halderman was charged with first-degree attempted grand larceny.
The book also details Letterman's feud with fellow late-night host, Jay Leno.
"Letterman assumed one day he would get a call from [Johnny] Carson or the head of NBC to offer him the job [as host of The Tonight Show]. It never happened," writes Zinoman, according to People.
"Rick Ludwin, head of late-night programming at NBC, respected Letterman as a great entertainer but was sceptical that he could draw the broad swath of viewers that made up the Tonight Show audience. He also saw Letterman as difficult to deal with. Whereas Leno was friendly and approachable, Letterman was distant, even hostile."
Leno's The Tonight Show routinely beat Letterman in the ratings.
