The 54-year-old hunk has already had an emotional moment with his newborn son too, as he revealed he shed tears while singing the tot a song.
He recalled: "I woke up this morning and I sang him House At Pooh Corner, the Christopher Robin [song]. I started crying. It was great."
John couldn't be happier after becoming a first time father, as he says everything is 'more exciting' now that he's brought new life into the world.
"It's more beautiful than I imagined. Everything is more colorful, more exciting. Music, you know, sounds better. He's just a beautiful little boy."
John announced the news of his son's birth on Monday in an Instagram post which featured Billy - who is named after John's late father - laid on his dad's chest.
The caption read: "From now on, the best part of me will always be my wife and my son. Welcome Billy Stamos (named after my father) #NotJustanUncleAnymore #Overjoyed (sic)"
Meanwhile, the former E.R. actor previously praised Caitlin - who he married in February - for dealing with her pregnancy "like a champ".
"Every day I'm with her I just realize, first of all, how lucky I am, but what a special human she is. She's magical. She floats into a room. I've never heard her say a bad word about anybody."