A source close to the show revealed: "Noel and Sandi are an odd combination and they certainly weren't top of the list - or even tenth on the list."
The insider then went on to claim that Love Productions, the creators of the original premise, are now concerned about what will happen to the show's popularity and cult following.
The producers are reportedly fearful that Noel, who is best known for his wacky sense of humour, will not translate with elderly viewers or school kids.
The Bake Off source continued: "A lot of people at Love Productions are worried about what Channel 4 is going to do with their show.
"Noel is going to turn off a lot of the family viewing that made Bake Off such a favourite."
MailOnline has contacted representatives of both Noel and the Bake Off for comment.
After months of speculation, Noel and Sandi Toksvig were confirmed as the new hosts of the beloved baking show on Thursday.
The trio will join expert new judge Prue, as well as show stalwart Paul Hollywood, when the series returns to screens later this year on Channel 4.