Abba couldn't believe his luck after Robertson let the ball bounce just outside the 18m box, enabling him to dart and pounce on to the skewed ball on the slippery surface.
Baxter charged out but his sliding challenge took Abba off his feet.
Olympic's new signing Van Elia pushed the ball from the penalty spot past replacement keeper Jarrod Hastings to level, 1-1.
Leftback Josh Anstis showed his displeasure at having to leave the field to accommodate Hastings.
The second-placed hosts drew first blood in the 4th minute when Josh Stevenson drilled the ball into the bottom-left corner of the net from the top of the box after the Rovers moved the ball around.
Robertson said first-choice 'keeper Jonty Underhill was too sick to be in the squad for Pacific Premiership gloveman Hastings to occupy the bench.
Striker Sam Messam was also a glaring omission from the Blues bench, Robertson saying it was his decision to "change personnel" in a 19-member competitive squad.
He echoed the sentiments of Olympic captain Mikey Halikias that the draw felt like a loss.
"We thought we were a much better team than them and we showed that in the first 20 minutes," he said, delighted to retain the O'Brien Challenge Shield and remain unbeaten at home.
The Rovers went 2-1 in the 60th minute from a Ryan Tinsley header, while striker Angus Kilkolly lay in the 18m box injured.
English import Ryan Tinsley had picked up a yellow card from Conger seconds earlier for getting involved in a verbal exchange between Chris Greatholder and Chas Lawrence.
In the 78th minute, Olympic drew 2-2 to a muted reception as they patiently worked the ball into the 18m box to curl it past a diving Hastings.
However, the Greeks should have done much better with one more player but seemed reluctant to attack or put the ball out wide.
"We're very, very disappointed with today's performance," Halikias lamented, adding it wasn't easy to win here.
He felt the penalty kick was deserved but Baxter's sending off was "dubious".
"We kept the ball and in the back we were solid enough so the extra man should have made it count but at the end of the day Napier set up well - 4-4-1 - so it was tough to break down."
Halikias said his troops lacked a bit of intelligence in the final third, with some bad decisions.
"We were very lucky with that second goal so Napier deserved that point."