Samoa kept France scoreless after halftime, but only just.
Milford, stepping off his left foot twice, beat three defenders to score five minutes into the new half.
France winger Jean-Philippe Baile was then held up by opposite Antonio Winterstein, and Samoa charged to the other end and standoff Pita Godinet scored.
In the last 10 minutes, France second-rower Sebastian Raguin knocked on a meter out, and the Samoans quickly turned the tables again, this time for Junior Moors to crash over for their fourth try.
Meanwhile, Scotland second-rower Dale Ferguson received a two-match ban from an international match review panel on Monday for a grapple tackle on United States hooker Tui Samoa, and will miss the quarterfinal against New Zealand on Friday.
U.S. hooker Joel Luani faces a one-match ban after being found guilty of a dangerous throw in the same game last Thursday. He has decided to appeal the ruling to try to be available for the quarterfinal against Australia on Saturday.
Tonga winger Jorge Taufua received a one-game ban for a dangerous throw in the 16-0 win over Italy on Sunday, but Tonga has been eliminated and he will serve the ban at his Manly club.