Cricket Australia has filed a complaint with Cricket South Africa after two days of sustained abuse of players' families by fans at Newlands.
Coach Darren Lehmann was clearly angry at the attacks which he labelled "disgraceful".
One fan in the members pavilion fronted David Warner as he left the field after being dismissed and proceeded to make comments, reportedly about his wife, as he walked up the race.
Warner stopped and had a short exchange with the man before walking on, but a single security guard who was present made no move to stop the man who continued to hurl abuse at the Australian.
It is estimated around 10 people were ejected for the abuse on day two and three were kicked out on day one for wearing t-shirts which attacked the Warner family.
"I think it's been disgraceful," Lehmann said.
"You're talking about abuse of various players and their families and personal abuse, it's not on at a cricket ground anywhere around the world, not just here, it shouldn't happen.
"You can have the banter, that's fine, banter is good-natured, fun by crowds but they've gone too far here. We've written to Cricket South Africa, Cricket Australia have done that, we'll see their response, but it's been poor. We'll see what happens, hopefully something."
Lehmann confirmed that the abuse by the fan toward Warner as he left the field was "personal and it was poor and he wasn't the only one".
"There's always going to be banter, as long as it doesn't get personal, but it has gone too far with the crowd here and they've got to be better than that when they're coming to international arenas to watch a game of two quality sides playing against each other," Lehmann said.
"They go hard on the ground, there's no doubt about that, but off the ground you don't expect that when you're leaving the ground or you're having a go at someone's family. It's just disgraceful.
"We accept it all around the world, but as soon as they cross the line and they talk about players' families the whole time and getting abused like that, it's just not on. there's been various incidents throughout the Test series but this one has taken the cake."
Lehmann admitted that Australian fans also go too far at times.
"That's not good enough from an Australian crowd point of view either," he said.
"We've just got to get better at watching the game of cricket, actually supporting both teams generally, and that's something that both boards have got to get around."
Elements of the fans have been particularly ugly at Newlands with reports they have lists of all the Australian players wives or girlfriends and are labelling them "sluts" or worse.
Fans continued their ugly behaviour as the team bus left the ground.
It has been a difficult series for Warner,
The batsman got into a verbal slanging match with Quinton de Kock after the South African used what the batsman called "vile and disgusting" about his wife Candice.
Two Cricket South Africa officials were then disciplined when they posed with fans wearing Sonny Bill Williams masks - a taunt directed at the Warners - at the game in Port Elizabeth.
The pair, marketing manager Clive Eksteen and communications manager Altaaf Kazi, have not apologised. They were recalled to Johannesburg and are not expected to be seen again.
The ground authorities had agreed not to let the masks but are understood to have been forced to back down when officials intervened.
Candice, her mother and their two children are in South Africa and have witnessed the distressing behaviour first hand.