Social workers also said the children's mother was "continually ignoring their advice against having him in bed with her" and rejecting advice about feeding.
In his conclusion, Judge Greene said litigation had begun after the older boy's leg was bruised when he was a few months old.
He said that one child was unintentionally injured by his father's use of "excessive force" and the other probably hurt as a result of the "rough, abrupt manner" in which he was handled by his mother.
He also concluded that the boys' mother "was ignoring advice against co-sleeping".
Judge Greene said evidence showed that the couple loved their children and would not deliberately hurt them. But he said evidence also showed the children's mother "always thinks that she knows best".
The judge said she was "brusque and physically forceful" with her children and dismissive of professionals.
He said he was concerned for the children's safety and that adoption was the only realistic option.