One woman, who identified herself as a person of colour called it a "hateful public display of what passes to you as humor (sic)."
Mr Camuglia also received some irate phone calls.
He told KRQE News he thought the sign was "a cute play on words."
"We didn't think anybody would be offended by that, it was not our intent to offend anybody," he said.
After the Facebook backlash, Mr Camuglia took down half of the sign and removed the Facebook post.
Though he still seems slightly miffed about the whole incident:
"I think if that offends some people, a statement about black olives, that somebody needs to re-evaluate their politically correct metre," he said.
He also posted a thank you to the restaurant's supporters on Facebook:
