Mr Parker said uncovered asbestos had been "piled up" near the workers' lunchroom and toilets.
He said workers on the site were "rightly upset" that they might have unwittingly carried asbestos fibres home in their clothes.
There would be no further work on the bridge until a meeting with RMS at 7.30am (AEST) on Friday, the union said.
RMS released a statement on Thursday confirming a "small amount" of asbestos was found during plumbing repair work on the bridge on Wednesday night.
The agency said samples of the potentially lethal substance were taken for testing.
"Given the location, there was no risk to the safety of workers of the public," RMS said.
It said a licensed contractor was removing the asbestos on Thursday.
- AAP