"If the IOC decides the games cannot go forward as scheduled in Tokyo you're probably looking at a cancellation."
Japanese government spokesman Yoshihide Suga, however, has said Pound's opinion doesn't reflect the official view of the IOC.
He told journalists there are no plans to cancel or postpone the Olympics and that it would go ahead as planned in Tokyo.
The coronavirus outbreak that began in China two months ago has infected more than 80,000 people globally and killed over 2,700, the vast majority of them in China.
But the virus has gained a foothold in South Korea, the Middle East and Europe, raising fears of a pandemic. Japan itself has reported four deaths.
The Olympics are set to open on July 24 with 11,000 athletes. The Paralympics open on August 5 with 4,400 athletes.
The modern Olympics, which date to 1896, have been cancelled only during wartime. The Olympics in 1940 were supposed to be in Tokyo but were called off because of Japan's war with China and World War II.
The Rio Games in Brazil went on as scheduled in 2016 despite the outbreak of the Zika virus.