New Delhi - Modi had earlier hit out at the Congress, saying parties seeking votes in the name of the architect of the Indian Constitution tried to erase his contribution to nation building.
Congress leader Rahul Gandhi, too, disapproved of Aiyar's remarks and said both he and the party expected the senior leader to apologise.
"He (Modi) is 'neech kism ka aadmi' (a vile man) who has no 'sabhyata' (civility)," Aiyar, who has often courted controversy with his off-the-cuff remarks said, attacking Modi.
The prime minister, while inaugurating the Ambedkar International Centre, had earlier on Thursday hit out at the Congress, saying parties seeking votes in the name of the architect of the Indian Constitution's name tried to erase his contribution to nation building.
"Such was his (Ambedkar's) incredible strength that when years after his demise efforts were made by such people to crush his philosophy, when efforts were made to erase his contribution towards nation building, his ideals could not be wiped out from people's memory," he said.
He said he would not be wrong in saying that "more people are still influenced by his (Ambedkar's) ideas than the family for which such efforts (to erase his contribution) were made".
Aiyar asked Modi: "What is the need for doing dirty politics on such an occasion?".
Modi, who was in Surat, latched on to Aiyar's comments, saying it was an insult to Gujarat.
"Mani Shankar Aiyar today said I belong to 'neech' caste, I am 'neech' (vile), this is an insult to Gujarat," he told an election rally.
"This is Mughal mentality which hates people who even wear good clothes," he said but asked BJP workers and supporters not to respond to Aiyar's remarks.
The Congress, too, strongly disapproved of Aiyar's remarks, with Rahul Gandhi saying he and the party expected the leader to tender an apology.
"BJP and PM routinely use filthy language to attack the Congress party. The Congress has a different culture and heritage. I do not appreciate the tone and language used by Mr Mani Shankar Aiyer to address the PM. Both the Congress and I expect him to apologise for what he said," Gandhi said on Twitter.