Workers renovating the Museum of Contemporary Art in Sydney were not protected from potential exposure to asbestos, the building union says.

The carcinogenic material has been found in the mortar of the sandstone building at Sydney Harbour, which is undergoing a A$50 million makeover. But WorkCover acting chief executive John Watson said allegations that asbestos was drifting around weren't "necessarily founded in fact" and that claims that up to 400 workers may have been exposed were alarmist.