Williamson made his total off 150 balls that included two boundaries and one six while Fulton's total came from three boundaries and two sixes off 189 balls before Gazi struck.
Ross Taylor was unbeaten on 31 at the break with skipper Brendon McCullum on 8 as Bangladesh was aiming for its first test win against New Zealand, having lost eight and drawn one of their nine matches previously played.
New Zealand, resuming on 117-1, looked comfortable batting on a pitch that seemed solid in slightly overcast condition. Play resumed 30 minutes early to make up for time lost because of rain a day earlier at the Zahur Ahmed Chowdhury Stadium.
Williamson, who scored 114 in the first innings, was at ease before he attempted a pull-shot off Gazi, rewarding a catch at forward short leg to Anamul Haque.
Gazi trapped Peter Fulton lbw earlier.
Bangladesh was dismissed on 501 in reply to the visitors' first innings total of 469.