"Emergency services attended the scene and a number of people were taken to hospital in a stable condition. They are our first concern and we are monitoring their progress," a spokesperson said.
Nine people were taken to hospital and about 100 staff worked in the Qantas office.
A St John Ambulance spokeswoman said seven ambulances were now at the scene.
They were being assisted by three St John managers and a search and rescue specialist "just in case".
Witness Leilani Sitagata said she saw a group of people cordoned off on the street.
"Nobody really knew what was going on," she said.
"From what I could see everybody looked a little bit flustered and a bit dazed".
A police spokesperson said they were called to a building in Victoria Street West by Fire and Emergency at 9.35am to assist with cordons.
Emergency services were responding to the effects of what was believed to be an unknown chemical substance, which forced the evacuation of the building.
AUT student Hayley Ware was caught in a police cordon as she walked by.
About 25 to 30 people had to stay in the cordon, some of them might have been from the nearby buildings but a lot of them seemed to just be passing by, she said.
Everybody held in the cordon seemed "pretty calm".
They were there for about 20 minutes before being released at the bottom of the street near the Queen St intersection.
Police say traffic is down to one lane on part of Victoria Street West.
The road is open for westbound traffic only between Queen S and Albert St.
The eastbound lane is closed from Albert to Queen streets.
A Fire and Emergency spokesperson said it had discovered that a smell of fumes was coming from the air conditioning system. Staff were working with contractors to find the source of the problem.
Firefighters were investigating to ensure that "it's not causing any harm to anybody".
