This morning he apologised and promised to "do better in the future".
In the tweets from 2013, sent before he entered politics, Taogaga asked: "Some kernels of truth though?"
Taogaga stood in the Helensville electorate in 2017 but lost to National's Chris Penk.
Prosser's column was sparked by having a pocket knife confiscated by airport security and was a pointed attack at Muslims.
The tweets were uncovered by The Nation and put to Prime Minister Jacinda Ardern in an interview.
Ardern said she wasn't aware of them.
"It is not the belief system of the Labour Party, but I am happy to reflect on a column from, again, seven years ago, and take any steps that might be required as a result in terms of making sure our candidates are familiar with the values we hold as a party," Ardern said.
In a statement issued to Newshub, Labour Party president Claire Szabo said Taogaga had been removed from the party's list.
"The opinions expressed in his tweets from 2013 do not represent the views of the Labour Party," she said.
"The Labour Party is committed to inclusion of all religions and stands against intolerance. After these tweets came to light they were discussed with Kurt and he has apologised."
Taogaga this morning offered a "wholehearted" apology for any offence the tweets had caused.
"In the intervening years, I have made positive efforts to educate myself and develop my thinking on the matter.
"I would like to re-state, firstly, to my Muslim friends and colleagues in the Labour Party and the community as a whole that I value you, I regret my comments and they do not reflect the views and values I hold today.
"I offered my apology to the party and my name has been withdrawn from the party list. I take full and one hundred per cent responsibility for those tweets from 2013. I will do better in future."
