Prince Andrew, Duke of York was summoned to Sandringham to face "strong words" from the Prince of Wales, a week after a similar meeting with his father.
Duke had to drive 140 miles to the Queen's Norfolk estate from his home at Windsor for the discussion on Monday, hours before a Panorama interview with Virginia Roberts Giuffre piled more pressure on the Duke.
It is thought the Prince, 71, wanted to reinforce the Queen's wish his 59-year-old brother step back from royal duties "for the foreseeable future".
The Duke of Edinburgh, 98, who has been staying at Home Farm, Sandringham, was also in the meeting to agree the move. A week earlier, with the Prince of Wales still on tour overseas, the Duke of York had been summoned for a meeting with his father.
"It was a tense meeting and Philip told his son he had to take his punishment," said a Sandringham insider.
"There was no screaming or shouting but Philip told him in no uncertain terms that he had to step down for the sake of the monarchy.
"Philip doesn't like trial by the media but he is realistic enough to realise that Andrew's actions were a danger to the very fabric of the royal family.
"It was agreed they would have another meeting with Charles. It was all very civilised but Charles calmly read him the riot act and told him there was no way back for him in the near future.
"Andrew thought he was being treated harshly as nothing has yet been proved... but he accepted the decision. He really had little choice."
"The feeling is that with Prince Philip retired at 98 and the Queen now 93, Charles has to take a firm grip on the family and plan for the future," said another source.
"The first task was to support the Queen in helping to sort out the Andrew problem."
It is believed the Duke left Sandringham after his meeting and did not stay to watch the Panorama interview at 9pm on BBC. During the hour-long documentary, Ms Giuffre alleged that at the age of 17 she was forced by paedophile financier Jeffrey Epstein to have sex with the Duke.
The Prince of Wales travelled back to London for Tuesday's Nato reception at Buckingham Palace for world leaders including Donald Trump.
Despite having been heavily involved in Mr Trump's state visit in June, the Duke was not invited to the reception.
The Prince had spent a week at Sandringham after returning from New Zealand and the Solomon Islands. It is thought he wanted to spend time with the Duke of Edinburgh, who has reportedly not been in the best of health.