"There was clearly inappropriate behaviour by some of the other guests and I deeply regret that this happened. I am extremely sorry for any offence that will undoubtedly have been caused."
But that was not enough and he was removed from his post as a parliamentary private secretary (PPS) to Transport Secretary Justine Greening after fresh allegations were made linking him to the behaviour.
A Conservative Party spokesman said: "Aidan Burley has behaved in a manner which is offensive and foolish.
"That is why he is being removed from his post as Parliamentary Private Secretary at the Department for Transport. In light of information received the Prime Minister has asked for a fuller investigation into the matter to be set up and to report to him."
Mr Burley's actions drew condemnation from British Labour MP John Woodcock, who chairs the party's Friends of Israel group.
Mr Burley, MP for Cannock Chase, issued an "unreserved, wholehearted and fulsome apology" over the party in a letter to the Jewish Chronicle earlier this week.
The controversy could damage Mr Burley's ministerial hopes, as private secretary roles are seen as a stepping stone to ministerial appointments in the British parliament.
The man in the SS uniform, reported to be groom-to-be Mark Fournier, could face charges under French law, which makes it illegal to wear Nazi uniforms and regalia unless for the purpose of a film, play or historical exhibition.