Patrick - who has two-year-old daughter Aurora with his wife Troian Bellisario - and Meghan, also 39, have been in touch a few times since 'Suits' wrapped, as the actor says they exchanged messages and "gifts" when their children were born.
But that hasn't stopped Patrick from feeling "pure fear" at the thought of getting in touch with Meghan, who has 16-month-old son Archie with Harry.
Speaking to the Radio Times, he added: "After our children were born, there were some texts sent and gifts sent, but I guess I'm a little scared. I think it's pure fear. I guess I'm scared about the idea of breaking through whatever walls exist to have that conversation."
Meanwhile, Patrick recently said he's "very happy" to hear Meghan and Harry have been "doing well" since they moved to the US with their son.
He said: "I miss my friend, but I'm very happy she's doing well. I don't really have any regular contact."
And Patrick also said he's pleased Meghan has been vocal about the upcoming US presidential election.
He added: "[I'm] very, very happy that she's becoming very vocal and doing whatever she can to help the election in November."