While he couldn't say who might be responsible, he said "there are people who are really passionate about their side and their views ... everyone has a thought and that's part of democracy".
If there were any attempts to intimidate voters at polling places, there were "mechanisms and protections in place for everybody at every polling place to make sure that doesn't happen", he said.
As for the number of people early voting, he highlighted the wide variation in voting laws between states.
"We have 340 million people, 50 states, and each state has different voting laws. So some states have allowed for what's called early voting where you can go in person ... others allowed in for mail-in ballots like I did - I requested a mail in ballot ... every state is different."
That variation could cause irregularities, he said.
Although there have been issues with those differing voting laws, he did not think they would impact the overall outcome of the election.
"But any time you have an irregularity or you are questioning the system and the way that people vote, you immediately need to fix it and make sure it doesn't happen again."
If the outcome on election night were not clear, Brown expected current US President Trump would do "the same thing that every other president did ... wait and follow the process".
Although Brown endorsed Donald Trump ahead of the 2016, he cannot do so this time around. The US' Hatch Act prevents him from "comment on any outcomes of the election and or who I supported".
Brown is returning to the US on December 20 to take up a new position at a law school.