He also apologised for any offence caused, but is happy that it started a conversation.
In the clip, Brown voices concern about how the director might treat the only black character in the film - Harmony, a prostitute who gets hit by her white pimp.
Damon interrupted Brown, saying they should pick a director based on merit, not diversity.
He then lectures Brown about how diversity works in Hollywood.
"When we're talking about diversity, you do it in the casting of the film, not in the casting of the show," said Damon. "Wow. Okay," replied Brown.
Twitter reacted with anger and the hashtag #Damonsplaining.
"Matt Damon is one of the most left wing actors he has been on front lines of all sorts of anti racist work its not even funny," Zaid Jilani tweeted.
- AP