They said PNG authorities were aggressively telling them to leave.
"The police, special forces, police squad are now in their hundreds, spreading through the prison camp and around the prison," Iraninan Behrouz Boochani tweeted from inside the camp.
"Navy soldiers are outside the prison camp. We are on high alert right now. We are under attack," he said.
"Immigration and police started searching the rooms and are saying 'Move Move' you only have an hour to move".
"Too much stress and tension here in Delta. Some refugees are crying."
"They are taking the phones and are very aggressive and are taking out some refugees who still remain in the rooms. Something terrible is happening right now, they are taking the refugees out of the rooms.
"They are destroying everything. Shelters, tanks, beds and all of our belongings. They are very aggressive and put our belongings in the rubbish bins. The refugees still are silent are watching them so scared."
At one stage, he said, he was tweeting from a toilet block as "they detroyed everything"
Fellow detainee Ezatullah Kakar tweeted the asylum seekers were peaceful, but scared, and "everyone is crying".
"WE WANT PEOPLE TO MOVE"
Mr Dutton told radio station 2GB: "We're very keen for people to move out of the Manus regional processing centre. I think it's outrageous that people are still there and they have trashed the facility, they are living in squalor, and the Australian taxpayers have paid about $10 million for a new facility and we want people to move."
"And obviously, it's in the end an issue for PNG police and the authorities up there but is an operation involving police in the centre this morning."
OPEN LETTER
The intervention comes after twelve former Australians of the Year penned an open letter asking Prime Minister Malcolm Turnbull and Opposition Leader Bill Shorten to immediately allow doctors access to asylum seekers in the facility.
The letter's signatories include Australians of the Year Rosie Batty, Ita Buttrose, Simone McKeon, Patrick McGorry, Mick Dodson, Tim Flannery, Fiona Wood, Fiona Stanley, Gustav Nossal, Peter Doherty, John Yu and Robert de Castella.
The group warns Australia's reputation on human rights is deteriorating because of the failure to meet the obligations of the UN Refugee Convention.
"We believe it is time to stop the unacceptable and internationally criticised treatment of the refugees on Manus Island, who, though innocent of any crime, have been incarcerated and now abandoned there," the letter says.
Russell Crowe has previously described Manus as Australia's national "disgrace", offering to take the refugees himself.
FEARS FOR SAFETY
The Manus camp was closed after a PNG Supreme Court ruled it was unconstitutional, and some 600 refugees were told to relocate to three nearby transition centres.
Around 400 of the asylum-seekers have refused to leave, saying they fear for their safety in a local population which opposes their presence on the island.
Canberra sends asylum-seekers who try to reach Australia by boat to detention camps in Papua New Guinea's Manus Island and Nauru, and blocks them from resettling in Australia.
The camps' conditions have been slammed by human rights groups, which have also campaigned to have them shut amid reports of widespread abuse, self-harm and mental health problems.
Canberra has strongly rejected calls to move the refugees to Australia and instead has tried to resettle them in third countries, including the United States.
