When a bystander tried to intervene, the driver turned his frustrations on her.
"Love, I know what I'm doing. Get out of the way," he can be heard saying.
The driver's employer Dean Crichton of Unique Carriage Hire defended the man's actions, telling 7 News:
"That is out of character but it was a very stressful situation.
"I don't condone his language but times like that help is better than negative comments."
Horse drawn carts were banned from the city's central business district after action from animal rights groups.
Tuesday the horse went back to work on Sunday, however with temperatures forecast to rise to the high 30s this week tourist attractions involving animals are likely to draw extra scrutiny.
"If this is how they treat the horses in public, just imagine what goes on behind the scenes," said the group Melbourne Against Horse-Drawn Carriages, which has urged for a total ban of the tourist carriages from Melbourne's roads.
"It's a horrible look. No one likes to see it we do our very best and the horses are very well looked after," said Crichton.
In October, the city of Rome was petitioned to ban similar horse-drawn attractions after a horse was filmed collapsing in the street. While a total ban was not enforced, botticelle carriages are no longer allowed to operate in temperatures of over 30 degrees.
