"He was found with... female undergarments. We don't know if these were the proceeds from the burglary today or a previous break-in," Acting Police Commissioner Stuart Cameron told Newsday.
Investigators took the panties into evidence for further forensic examination.
The commissioner said it was unclear whether Cicale knew the woman whose panties were raided or not.
Neighbours and colleagues expressed shock at the arrest and said the alleged behaviour would be totally out of character for the respected judge.
Cicale, a Republican, was elected to the bench in 2015.
He has also been East Islip's town attorney since 2012 and worked for the Legal Aid Society, which represents indigent defendants.
"He has a reputation not only as a good judge but as a good human being, too," said Chris Brocato, president of the Suffolk County Criminal Bar Association.
Neighbour Jay Moceri, a 42-year-old construction worker, told the Daily News that burglary would be "something definitely out of character" for Cicale.
"He's always out there playing basketball with his kids and seems to be a good family man," said Morceri.
Cicale was held overnight by police and is scheduled to be arraigned in his own Suffolk District Court in Central Islip on Friday.
A Nassau County judge will preside at the arraignment to avoid conflict, according to a spokesman for the state Office of Court Administration.