Massive construction projects that transformed the face of the entire city were carried out at break-neck speed and much of that infrastructure remains to this day.
The shinkansen, or bullet train, took off from Tokyo bound for Osaka, for the first time on October 1, 1964, just nine days before the Olympics.
Yet the Japan of today is a very different place.
Decades of economic stagnation has left the public with less tolerance for extravagant, over-the-top construction projects and the waste of public funds.
Although most Tokyoites seem to support the 2020 games, many want to see a more streamlined, sensible and economic approach taken to building Olympic venues.
The construction of a new National Stadium - the main athletic arena for the games - has become a flash-point of sorts where this old, build-it-big approach is clashing with a new, more sober and subtle conception of how to develop an urban environment.
The new venue will be built at the same site used for the 1964 games - the Meiji Shrine Outer-Gardens or Meiji Jingu Gaien - and the current stadium is marked for demolition later this month.
The replacement, designed by renowned British-Iraqi architect Zaha Hadid who created the London 2012 Olympics Aquatics Centre, is remarkable for its size and peculiar design.
Its unusual shape has been likened in media reports to a gigantic cycling helmet, an alien space ship, and even a vulva.
Edward Suzuki, one of a number of high-profile Japanese architects campaigning against the new stadium, says it is not the design of the building to which they are opposed, but the location and context.
He says if the proposed site for Hadid's design was the waterfront area in Odaiba - a large artificial island in Tokyo Bay - for example, it wouldn't be an issue, as there is lots of land and very few trees.
"The Meiji Shrine Outer-Gardens, on the other hand, has been cultivated for almost 100 years to have the green that we have today.
"They are going to do away with so much green and build this very artificial man-made structure, which is way too large for the context.
"It is not to human scale. It is so overpowering and so overwhelming," he says.
In addition, Suzuki says the stadium side walls, which in places will be up to around 25 storeys high and 100m long - will come literally right up to the edge of surrounding footpath.
"Not only is it an eyesore, but a psychological threat - a monster," he says.
Objections to the new stadium are not just limited to its physical dimensions and incongruence with the surroundings.
Not surprisingly, the exceedingly high cost is also an issue, although the Government has taken steps to try to reduce the burden on taxpayers.
Originally, the total cost of the project was 300 billion (NZ$3.48 billion), but that was cut back to just under 170 billion in June when the Japan Sport Council tabled a new proposal in the wake of public outcry over the stadium.
The new proposal also reduced the overall space of the stadium from 290,000 to 222,600sq m and lowered the tallest point of the stadium from 75 to 70m.
Suzuki says these new figures are unrealistic and don't go far enough.
"They reduced the stadium size by 20 per cent and by that they magically reduced the cost by 50 per cent. It is only a schematic or preliminary design at this stage. There is no way they can say they will keep the cost at 170 billion," he says.
He says the Government estimate is based on last year's values for unit prices of construction, which have since gone up and are predicted to increase further - in part due to the hype surrounding the Olympics and the reconstruction boom in the tsunami-ravaged Tohoku region.
In addition, consumption tax also increased from 5 per cent to 8 per cent in April this year.
"I wouldn't be surprised if the final cost, again, comes up to 300 billion if not more," says Suzuki.
Another high-profile Japanese architect, Fumihiko Maki, shares Suzuki's concerns about the project's cost.
Maki, who won architecture's top award, the Pritzker Prize, in 1993, says that, in addition to the excessive price of construction, the running costs for the stadium - an estimated 4.6 billion a year - are way too high.
But the Japan Sport Council has defended this high ongoing expenditure by arguing that the stadium will be well used after the Olympics for a range of activities, not just sports, but also for musical and for cultural events.
Yet Maki believes this is wishful thinking and says that, realistically, the demand for a stadium that seats 80,000 people for rock concerts and similar cultural events will be very low.
"In reality in Japan we are losing population. We are not an upcoming nation like India or China. So, we should be more modest in our thinking, even for such cultural events," he says.
In addition, he says it is not an IOC requirement to have a stadium which seats up to 80,000 people and that the Government decided on this number not for the Olympics, but for the Rugby World Cup.
The 2019 Rugby World Cup will be the first major, international event held in Tokyo's new National Stadium and is being viewed as a dry run for the Olympics held the following year.
