Videos allegedly shot in the Mosul area appear to show Iraqi security personnel executing a detainee and brutally beating others, Human Rights Watch said.
Iraq declared victory over Islamic State in Mosul earlier this week after a nearly nine-month battle that ravaged the city and took a heavy toll on residents and security forces.
The videos "appears to show Iraqi soldiers and federal police beating and extrajudicially killing detainees", the rights group said in a statement that included links to clips, which were posted on Facebook.
In one video, men in Iraqi army uniforms beat a bearded detainee, drag him to the edge of a cliff, throw him off and shoot him and another body at the bottom.
HRW said it had located the site of the first video - which appears to have been removed from Facebook but was circulated elsewhere online - using satellite imagery, but was not able to confirm where three other clips were filmed.
Those videos, which show men in army and police uniforms beating detainees, were still viewable on Facebook, where they were posted by a man HRW said "regularly publishes information regarding security and military activities in and around Mosul".
Speaking to the BBC, Belkis Wille, senior researcher at Human Rights Watch, said: "In the final weeks of the battle for west Mosul, the pervasive attitude that I have observed among armed forces has been of momentum, the desire to get the battle wrapped up as quickly as possible, and a collapse of adherences to the laws of war."
This was accompanied by "what seems to be a resulting decline in their respect for the laws of war", she said, calling on Prime Minister Haider al-Abadi to launch investigations of abuses.
According to the BBC, an interior ministry spokesman said that if the video - purportedly filmed in recently-liberated Mosul - was genuine then the soldiers involved will be brought to justice.
One man, Falah Aziz, claimed to have beheaded 50 Isis fighters and stressed he was completely "at ease" doing so.
The video of the man being thrown from the cliff, which was posted to the Mosul Eye blog yesterday, comes just days after a US-backed coalition of Iraqi forces announced it had freed Mosul, the city in which Isis proclaimed its caliphate in 2014.
But small-scale fighting is still going on inside the Old City, with an unknown number of Isis troops - perhaps as many as hundreds - hiding out.