'Harm done to all' in Labour sex assault claim review: PM