She reflected: "I've never felt that I was much older than Harry anyway. I still feel 18 and probably act that way half the time - all those years in kids' telly probably didn't help.
"Although it was just a bit of fun we decided it was best to keep it to ourselves, as we were both working for Simon (Cowell). We were both single, we got on well and we had a laugh."
However, when their relationship became public knowledge, things took a dramatic turn for the worst.
In her book, serialised in The Sun on Sunday newspaper, she said: "It began to go wrong when Harry was pictured coming out of my house one morning. And once that was out, it was open season.
"After that anyone could say anything. In the street people started shouting at me 'paedophile' and 'pervert'. A One Direction fanzine had me as a voodoo doll, with arrows (pins) pointing to bits of my anatomy."
Flack also claims she had a brief romance with Prince Harry.
She revealed that she and the royal got close after meeting through a mutual friend, however they called off their relationship after it was made public.
"I knew my friend was friends with Prince Harry, and I'd never met him, so I thought, 'Oh, that's quite exciting,' and for a moment I perked up. So I was just sitting there and he arrived with a few others in tow and we all spent the evening chatting and laughing.
"[But] once the story got out, that was it. We had to stop seeing each other. I was no longer Caroline Flack, TV presenter, I was Caroline Flack, Prince Harry's bit of rough."
For more gossip and celebrity news listen to ZM Spy:
- Bang! Showbiz