That clashed with the stance of Sajid Javid, the Home Secretary, who had insisted last week that he would "not hesitate" to prevent the return of anyone who supported terrorist organisations abroad.
With coalition forces preparing to declare victory over Isis and America's troop withdrawal from Syria imminent, there is intense concern in Washington that "time is running out" to bring the terror group's fighters to justice.
Many are being held by the US-backed Kurds, a stateless group that have demanded Britain and others take the fighters off their hands.
One well-placed US government source said that Britain's refusal to take back UK jihadists was effectively a policy of "leave them at large and hope they don't find a way back".
Another said that European allies had been warned their countries would be at risk if captured jihadists "are not locked up for a long time".
There are also concerns that any Isis fighters who reach Europe could make their way to America undetected because of the visa waiver scheme.
While suspected terrorists should be on watch lists that would stop them boarding a plane, US officials acknowledge the system is not "foolproof".
A senior Government figure defended the position, insisting that Britain wanted jihadists to face justice in the Middle East.
Ministers also fear they would struggle to secure prosecutions because of the legal difficulties in bringing fighters back to the UK and the make-up of current laws. Sources told the Sunday Telegraph that at least seven British men are among the foreign fighters detained in Syria.
The declaration of victory over Isis' caliphate appeared imminent last night as its fighters were pushed back to a pocket of land just 700 sq m in eastern Syria.
Donald Trump, the US President, had forecast a final defeat would come yesterday, but Syrian Democratic Forces commanders slowed a push on the village of Baghouz over fears that civilians were being used as human shields.
The end of Isis' territorial control does not mean total victory over the terror group, which still has thousands of fighters and several sleeper cells.
However, it is the moment of success that Trump has been waiting for, with the withdrawal of America's 2000 troops expected to follow, possibly as early as April.
The decision, which was made before Christmas and both infuriated allies and triggered James Mattis, his Defence Secretary, to resign, has created looming uncertainty.
Mike Pence, the US Vice-President, yesterday attempted to calm fears at a security conference in Germany, saying: "The US will continue to work with our allies to hunt down the remnants of Isis wherever and whenever they rear their ugly heads."
But the rift with allies was clear.
Angela Merkel, the German Chancellor, said: "Is it a good idea for the Americans to suddenly and quickly withdraw from Syria? Or will it once more strengthen the capacity of Iran and Russia to exert their influence?"
Discussing Begum's future, Gauke said: "Obviously we have to act within the powers that we have.
"It is the case we can't make people stateless, but without getting to drawn into the specifics, the approach that we take as a government, which is the responsible one, is to ensure that we protect the British public. That is the key thing."
