Mallard shot back, saying the union should have better things to worry about than the private expenditure of New Zealanders while overseas.
"Part of our group decided to partake in a tourism activity that involved a two-hour climb up a mountain to have a chance to look at gorillas. No taxpayer funds whatsoever was spent on that experience.
"The funders of the Taxpayers' Union might want to consider whether it's their role to comment on the private expenditure of Kiwis aboard. Surely they have something better to do?"
Former Speaker and National MP David Carter also jumped to Mallard's defence.
"New Zealand doesn't have strong links to many African countries, and there are trade opportunities for us.
"They are an opportunity for other MPs to see for themselves how other democracies work ... All in all I thought they were of tremendous value, and far from a junket."
Carter added that taking in a gorilla tour while on one day off in a fortnight "doesn't sound extravagant to me".
Mallard posted a video and photos of the gorilla tour on Twitter yesterday.
The Speaker's delegation is the first such trip to Africa, and Mallard has said it could lay the groundwork to boost Africa-NZ trade in the decades ahead.
The budget for the trip is $164,000.
The leg of the trip to Ethiopia and Rwanda included visits to the African Union, the UN Economic Commission for Africa, and a rehabilitation centre that receives New Zealand and United Nations aid money.
The group, including National MPs Nick Smith and Nicky Wagner, Green MP Gareth Hughes and Labour MP Angie Warren-Clark, has spent about a week in Ethiopia and Rwanda, and is now in Turkey for Anzac Day commemorations.
The Turkey leg of the trip has taken on renewed importance since the March 15 terrorist attacks in Christchurch; Turkish president Recep Tayyip Erdogan used the gunman's video at election rallies and then, referring to the Anzacs at Gallipoli, said that anyone visiting Turkey for anti-Muslim reasons would be returned "in coffins", as their grandfathers were.
Deputy Prime Minister and Foreign Minister Winston Peters travelled to a special meeting of the Organisation of Islamic Co-operation in Istanbul to assure Muslim leaders that New Zealand was a country that embraced its Muslim community.
Mallard told the Herald before leaving on the trip that the delegation would meet Erdogan in Turkey.
"My message will be that all of us are very concerned about the terrorist attack in Christchurch, and I hope the way it's been handled in New Zealand and internationally will be useful to the world in a more peaceful future."
He said the trip to Africa was his own initiative and followed advice that New Zealand should establish more frequent relations with African countries.
"The advice when I first came to be Speaker from [former Deputy Prime Minister and former Foreign Minister] Don McKinnon was that there are lots of places where New Zealand has links only when we want something.
"One of the roles of the Speaker can be to develop or reinforce links which otherwise would be very thin. There's no doubt in the longer term, because the GDP is increasing quite rapidly in a number of African countries, good contacts now make a lot of difference in 20 to 30 years as far as trade is concerned."
According to the Ministry of Foreign Affairs and Trade, New Zealand exports to Africa have more than tripled in the decade to 2014, when two-way goods trade was worth almost $2.5 billion.
In Turkey, Mallard will deliver an address and lay a wreath at the Anzac dawn service. The delegation will also travel in Ankara to meet Mustafa Sentop, the Speaker of the Turkish Grand National Assembly.
