Tony Tiopira gave him the fingers and yelled back an expletive-filled response that include the phrase: "you f**king honky", Mr Brown said.
He approached the Tiopiras and Ricky grabbed his shirt and punched him before his son started "laying into" him too.
Mr Brown told the authority Ricky Tiopira "got right into my face and he was going apeshit".
He said he rang his father and told him to bring his guns.
A disciplinary meeting was arranged by the orchard's managing director which the Tiopiras failed to attend, the authority found.
Following an investigation the company determined their actions amounted to serious misconduct.
"The incident on 2 September 2012 has left fellow co-workers concerned and fearful," their dismissal letters stated.
"The threat of violence seriously undermines the trust and confidence within you as an employee."
The Tiopiras said their dismissal was unjustifiable and claimed not to have received their dismissal letters or the letters inviting them to a disciplinary meeting.
The authority sided with the orchard but dismissed the application for damages, finding "the evidence produced to the authority in support of this claim is rather sparse".