Roger Woodgate has lost his bid to get a restraining order against neighbours Margaret and Craig Harris who sparked a media frenzy by suggesting he was the long-lost Lord Lucan.
Accusations flew at the hearing in Marton District Court on Friday.
"He's a freak control," Mrs Harris bellowed, pointing her finger fervently at Mr Woodgate from the witness box.
"What are we supposed to do fly past his place on a broomstick?"
Mr Woodgate retaliated by calling the Harrises "nutters".
Judge Bernard Kendall threw out Mr Woodgate's application for a restraining order against his neighbours after a three-hour hearing.
The reclusive Englishman stood in disbelief after the ruling was made.
"What am I going to do now?" Mr Woodgate asked. "I don't want to have to pack up and move because of a family of nutters, but now they're free to do what they want and no one's going to stop them.
Giving evidence in court, Mr Woodgate detailed more than 30 incidents of alleged harassment at the hands of the Harris family since he moved to his property near Marton in 2002.
Many of those involved occasions where members of the Harris family gestured aggressively at him as they drove by.
Others included their "telling of lies" that led to him being fingered as suspected murderer Lord Lucan in August.
His face was splashed across the world's newspapers - along with pet possum Redfern - and his life put on display for all to see.
He had "played along" with media, he said, because he did not want to be depicted as a crazed hermit, and he wanted to tell his side of the story.
The Harrises' lawyer David Woodbridge said the family had never been seen to physically harm Mr Woodgate, or enter his property.
Mr Woodgate's lawyer, Mike Ryan, questioned whether Mrs Harris would ever cease her attempts to paint Mr Woodgate as Lord Lucan.
Mrs Harris, banging her fist against a wooden ledge in frustration, said it would take a DNA test for her to let it be.
Judge Kendall said he would not grant the restraining order as he did not believe one was warranted.
"There is a lingering anxiety that an order, if made, will in fact be used as a weapon and not a protection," he said.
Mr Woodgate said an appeal was unlikely.