The man said the referee then attempted to send those players off the field.
"But they refused to leave and continued to abuse him [the referee].
"He had to abandon the game and left the field himself."
Ellerslie AFC chairman Tim Adams confirmed the situation got so tense the young referee was concerned for his safety.
"He locked himself in the changing rooms and called the development officer which was the right thing to do."
It is understood the development officer made sure the referee was safe and could leave the venue.
Manukau City club secretary Maia Fowler said there had been an incident and the team chose to withdraw.
She said the issue had been caused by other non-club members playing in the team.
It is understood the case is now with the federation's disciplinary committee.
Another team, the Ranui Swanson Football Club, was forced to withdraw due to what was described as a "serious incident" following their June 15 match.
In a statement announcing the ban, the NFF said the West Auckland club was already on its final warning after a previous serious incident on May 2.
The incident that led to the ban occurred following a Mens' Conference match with AFF team Glen Eden Rangers.
A witness at the match said a "member of management" from the Ranui club accused the referee of "being a cheat". Glen Eden won the June 15 match 1-0.
The witness said there was nothing discriminatory in the comments but "what happened wasn't acceptable, 100 per cent".
"It was an individual who wasn't happy, and was angry with a decision that happened during the game," the witness said.
A spokesperson from the Swanson Football Club said she did not know how long the ban was in place for.
"We haven't been told how long it is and we are meeting with the NFF next week to try to resolve things"
The woman confirmed a non-playing member had committed an R13 offense at the completion of the game.
An R13 offense includes insulting, abusive language and/or gestures against a match official.
Ranui Swanson is appealing the decision by the NFF.
A statement from NFF said: "The withdrawal from Competition will remain in place until such time as NFF is satisfied that appropriate steps have been taken by the Club, and that the players, coaches and all persons associated with the Team are clear on NFFs expectations and the consequences of further transgressions,"
In the May 2 final warning issued by the NFF to Ranui Swanson, it was stated the club would be — if it continued to breach code of conduct — "withdrawn from all NFF Competitions immediately and entry for 2020 season would not be automatically accepted without fuller consideration."
The two withdrawals come as officials continue to crack down on abuse against referees.
There have been a handful of refs physically abused by players in the past year, although most were during rugby games rather than football.
Last June, a 20-year-old Oamaru man was charged with assault after allegedly punching the referee at the end of a Citizens Shield North Otago rugby semifinal in Oamaru.
He was given a five-year ban from the sport at a judicial hearing.
And in May, a senior rugby player who punched an 18-year-old Wellington referee copped a 10-year ban.
Masterton Marist player Sautia Lemalu struck Alex Blakeway during his team's senior reserve match at Masterton's Memorial Park.
