Buckingham Palace declined to comment on the visit.
The Sun reported earlier that the Sussexes were forbidden to bring a photographer to Windsor Castle to snap the moment the Queen met Lilibet.
They were kept at a distance by senior royals for the rest of the Jubilee. They flew back to California last Sunday before celebrations wrapped up, with Harry soon spotted playing polo again back home.
Meanwhile, William and Kate took centre stage at the Jubilee pageant, with their youngest son Prince Louis stealing the show.
It's expected that the Duke and Duchess of Cambridge will move to a house on the Windsor estate this year. Until recently, the newly refurbished Adelaide Cottage was thought to be their first choice for when they make the move from London to Berkshire.
But the Sunday Times reported that the Cambridges could move into the "big house", since Prince Charles plans to spend less time than the Queen at Windsor Castle when he becomes king.
And disgraced Prince Andrew could be sent off to Scotland out of the public eye after secret talks on his future were held by his mother the Queen.
The monarch has reportedly spoken with the Duke of York as well as other senior royals about how he can rebuild his life away from public attention.
