Diana's brother is worried that people who watch The Crown will "forget that it is fiction".
The royal drama portrays Princess Diana in the fourth season, but Charles Spencer told Love Your Weekend with Alan Titchmarsh that the series is full of "a lot of conjecture and a lot of invention", reports People.
Spencer said, "You can hang it on fact, but the bits in between are not fact."
Asked if he felt uneasy watching the show, Diana's brother admitted that he did "a bit".
"Actually, The Crown asked if they could film at Althorp [the Spencer family home], and I said obviously not. The worry for me is that people see a programme like that and they forget that it is fiction. They assume, especially foreigners, I find Americans tell me they have watched The Crown as if they have taken a history lesson. Well, they haven't."
And Spencer feels "very passionately" that it's his responsibility to "honour her memory".
"I feel it is my duty to stand up for her when I can," he said.
"She left me, for instance, as guardian of her sons, so I feel there was a trust passed on. And we grew up together. If you grow up with somebody they are still that person — it doesn't matter what happens to them later."
But the show's writer as well as the cast have insisted that the show dramatises real events within the royal family. Emma Corrin, who plays Diana, said she understands the criticism.
"I understand why people would be upset because this is history. And even with Diana, it's still very much fresh, everything that happens," the 24-year-old actress said on the Tamron Hall Show.
"So I do really understand if people would be upset."
Spencer recently accused the BBC of a "piecemeal apology" for using fake documents to help land Diana's famous interview with journalist Martin Bashir 25 years ago.
He told People that the BBC "have yet to apologise for what truly matters here: the incredibly serious falsification of bank statements suggesting that Diana's closest confidantes were spying on her for her enemies".
"This was what led me to talk to Diana about such things. This in turn led to the meeting where I introduced Diana to Bashir, on 19 September 1995. This then led to the interview," he continued.
"The BBC have so far refused to acknowledge the above. They claim Diana wasn't misled. They have ignored my inquiry as to whether the apology over their false bank statements extends to the ones that actually persuaded Diana to meet Bashir."
The BBC earlier announced that they had hired former British Supreme Court Judge Lord John Dyson to head an independent investigation into Bashir and his tactics.
And Prince William welcomed the investigation, saying it was a "step in the right direction".
"It should help establish the truth behind the actions that led to the Panorama interview and subsequent decisions taken by those in the BBC at the time."