Then, realising his mistake, Simon issued an apology for his comment and said it's why he "should never do live TV".
Simon said: "I apologise. I shouldn't have said that. That's why I should never do live TV."
Rylan also downplayed the incident after the show, saying it was a "joke between friends".
The controversy comes shortly after it was announced Simon will continue as a judge on America's Got Talent until 2019.
Read more on Spy:
• Antonia Prebble's relationship split
• Why ex Breakfast host is 'happier' now
The Brit is one of the co-creators of the show and has proved a hit with his first stint on the panel this year, so NBC bosses have extended his contract for another three seasons.
Paul Telegdy, NBC Entertainment alternative and reality group president, said in a statement: "It was terrific to see Simon in his judge's chair this past season, bringing his unique expertise and enthusiasm to the show, night after night.
"His brilliance has made America's Got Talent the industry leader that it is today and his ability to evaluate and spot talent is unmatched, so we couldn't be happier to have him seated at the judge's table for the next three years."
- Bang! Showbiz