Skipper Michael Clarke was bowled offering no shot as Anderson got one to nip back off the seam.
Steve Smith and Rogers added 48 for the fourth wicket but with the score on 110, Smith was caught at second slip off Broad's bowling.
Rogers fell two runs later.
Smith, Watson, Rogers and George Bailey all average under 35 in the series.
Bailey's scoreless 19-ball knock ended with a controversial caught-behind dismissal at 6-122.
Umpire Aleem Dare gave Bailey not out but TV umpire Billy Bowden overturned the decision despite replays showing no edge on HotSpot and a possible noise on the Snicko as the ball passed the bat.
Brad Haddin successfully challenged an lbw decision against him on 35.
However, tailenders Johnson and Ryan Harris, who had scored half-centuries earlier in the series, were unable to reach double figures in front of the bumper crowd of more than 78,000.
Peter Siddle was out for a duck in the final over before stumps, with Haddin unbeaten on 43.
Thirteen wickets fell in the day's play, including Australia losing 6-68 in the third session.
Rogers said Australia would have to fight hard to get back into the game.
- AAP