He reportedly realised his mistake but as he couldn't see anyone on the platform, he decided to take off.
But it turned out there were people on the platform who intended to catch that train, and they were forced to wait a whole six minutes for the next service.
Those passengers complained, which prompted the train's operator, West Japan Railways, to issue a solemn apology.
"The great inconvenience we placed upon our customers was truly inexcusable," the company said, according to Sora News 24.
The company said staff would get extra training so the mistake wouldn't happen again.
This isn't the first time this has happened in Japan, where the punctuality of its intricate train networks is taken extremely seriously.
In November, another train company formally apologised to passengers who were inconvenienced when one of its trains left the station 20 seconds early.
The train was meant to leave Minami Nagareyama railway station station at 9.44.40, but it actually left at 9.44.20.
No one complained in that instance, but the company still issued a red-faced apology.