According to The Sun newspaper, Simon, 52, is also worried about how the new show will affect his X Factor when it returns to screens later this year.
The insider continued: "The big fear is that viewers will start seeing The Voice as the leading singing show.
"That would mean the best singers flocking to apply to it instead of The X Factor - and these shows live or die on their talent.
"Simon always learns lessons. He will be seriously considering whether he has to be nicer to contestants.
"He has softened his Mr. Nasty image over the years. But maybe now he needs to turn fully into Mr. Nice. It's clear viewers are starting to get a taste for more positive shows."
However, his spokesman insisted: "Simon is happy with the ratings. The judges are looking forward to the live shows."
- BANG! Showbiz