Hughes dismissed the Swenson indictment, faulting the prosecutor's mistakes.
Then he said something else that eventually became the basis of the rebuke.
"It was a lot simpler when you guys wore dark suits, white shirts and navy ties," Hughes said, according to the 5th Circuit. "We didn't let girls do it in the old days."
The US attorney's office appealed the case's dismissal, which set up a panel of appellate judges to send back its censorious response.
In reinstating the Swenson case, the appellate court also took an unusual action: It ordered Hughes, now 76, to be replaced with a different judge.
More than a third of the 300,000 members of the American Bar Association are women, according to a recent report from the ABA's Commission on Women in the Profession, and female litigators have been subjected to male colleagues' sexist comments for decades.
Two years ago, the ABA adopted an anti-discrimination resolution in its rules of professional ethics, making harassment and discrimination grounds for findings of misconduct.
"It's important to set up a process and culture that doesn't marginalise people who legitimately say, 'You did this to me,' " said Michele Coleman Mayes, former chair of the ABA's Commission on Women in the Profession.
The resolution and the subsequent arrival of the #MeToo movement eased the way for women to come forward with claims of sexual harassment or unfair treatment, Angela Brandt, president of the National Association of Women Lawyers, said. Still, she said, female lawyers are hesitant to do so.
"It's a male-dominated profession," she said. "Women feel weak highlighting this type of bad behaviour, and, in certain circumstances, it can jeopardise their careers."
Experts say the industry has therefore faced less scrutiny than other professions - though there have been notable exceptions: Judge Alex Kozinski, for example, resigned from the federal bench last year amid allegations of sexual misconduct. The accusations led to a review of how the judiciary handles claims of harassment, though Kozinski's abrupt retirement cut short any investigation into the merits of the allegations against him.
Hughes, who has been on the bench for almost three decades, denied the accusation of sexism. He told the Post that his comment was not directed at the prosecutor in his Houston courtroom but at an "inappropriately dressed" woman.
The 5th Circuit acknowledged that Hughes's comments may have been intended for someone other than the US attorney. Nevertheless, Judge Edith Brown Clement of the 5th Circuit wrote, as Hughes "excoriated the prosecution," he "attributed the Government's mistakes to the prosecutor's sex."
The female prosecutor was not named in the 5th Circuit's decision, but Assistant US Attorney Tina Ansari has been prosecuting the Swenson case since 2015, according to the Justice Department.
Ansari, who attended the University of Houston Law Centre and has been practicing for more than 15 years, could not be reached for comment.
Hughes denied favouring either sex. "After I've made it through my career with a woman's first name, of course I'm going to be sympathetic," he told the Post.
