The Sussexes are in the UK this week for two charity events, and have reportedly asked for a private security firm to protect them with a "ring of steel" throughout the trip.
News of the declined invite was revealed when Charles was snapped heading to church in Balmoral alone over the weekend.
It comes as the royals, many of whom have spent time in Balmoral with the Queen during her summer break, brace themselves following the arrival of the Sussexes in the UK.
Sources close to the Prince of Wales say he "loves and misses" Harry, Meghan and his grandchildren, and feels wounded by their words and actions since spending time with them at the Queen's Platinum Jubilee in June.
Tensions were heightened last week after Meghan's interview with The Cut during which she suggested she and Harry were forced to leave the UK because "by existing, we were upsetting the dynamic of the hierarchy".
The pair are set to arrive in Windsor today after reportedly landing in the UK yesterday.
A potentially awkward reunion with Prince William and Kate Middleton is thought to be unlikely, in spite of the fact that they will be staying just a few hundred metres away.