Westpac Group Executive, Consumer, George Frazis, said in a statement that it understood "the 'foreign ATM' fee has been deeply unpopular with consumers."
NAB Chief Customer Officer of Consumer Banking and Wealth, Andrew Hagger, said the decision was about making banking fairer.
"We know it has been frustrating for them to be charged to withdraw their own money from an ATM, and the change we are announcing today will benefit millions of Australians."
Reserve Bank of Australia data shows Australians made more than 250 million ATM withdrawals from banks other than their own last year.
Australia's fifth largest bank, Macquarie, said it did not charge ATM fees and would refund the A$2 fee if their customer was slugged by another bank.
The fee abolition will not apply to cards from overseas banks.
The Australian Bankers Association said Sunday's announcements were another example of how banks were working to improve their services.
"This is the latest in a suite of initiatives by banks to create better products and services for customers and boost customer choice, including reducing interest rates on credit cards and offering fee-free transaction accounts," CEO Anna Bligh said in a statement. "A competitive banking system is good for customers and good for the sector."
But the changes, while welcomed by federal politicians, did not stop calls for a banking royal commission.
"Imagine how we could get better banking for all Australians if we had a banking royal commission," Opposition Leader Bill Shorten said.