The latest move is indicative of the strength of feeling among other royals about his "pariah" status.
'For his own good'
A source told the Sun that Prince Andrew was told to stay out of sight today "for his own good" following the tense family talks.
The Duke is thought to have challenged the ban but to no avail.
A source close to the Duke insisted that it was his own "personal decision" not to appear in public.
Having been forced to pull out of the service of thanksgiving at St Paul's Cathedral during the Platinum Jubilee celebrations earlier this month after testing positive for coronavirus, he was looking forward to the public appearance and considered a small step on the way to his return to some form of royal life.
He will still join the Queen and rest of his family for the investiture in the castle's Garter throne room and the traditional lunch that takes place afterwards in the Waterloo Chamber.
But neither the Duke nor the Queen are expected to join the public procession and church service due to ongoing mobility issues.
Garter Day is one of the highlights of the Royal family's summer schedule and is taking place this afternoon for the first time in three years.
The Duchess of Cornwall will be among those invested into the country's oldest and most senior Orders of Chivalry alongside Sir Tony Blair and Baroness Amos.