"When it comes to football, this has been the hardest decision I ever had to take in my career," said Koller, who has been publicly backed by his players after missing out on the World Cup.
Captain Christian Fuchs said he wanted to continue working with the coach and striker Marc Janko posted an open letter to Koller on his Facebook page, saying all the players wanted him to stay.
In 18 matches under Koller, Austria had eight wins and seven losses. Austria next plays the United States in a friendly in Vienna on Nov. 19.
"I am very glad the decision panned out this way," Austria federation president Leo Windtner said. "I see this as a positive development for the team as well as for the federation and the fans. I am sure this team under his professional leadership has very good chances to reach the next European Championship."
The central board of the Swiss federation said last week that Koller was its main candidate to succeed Hitzfeld, who will step down after the World Cup in Brazil. The board confirmed it had spoken to the coach without reaching an agreement.
"Now we have clarity," Swiss delegate Peter Stadelmann said. "We will continue our search for a successor and will talk with other suitable candidates ... We are not in a hurry as time was only a factor in regard to Marcel Koller."
Koller, who spent his entire career as a player with Grasshoppers in Zurich, played 55 times for the Swiss national team. He coached FC Wil, St. Gallen, Grasshoppers and Bundesliga sides Cologne and Bochum before moving to Austria.