It likely came from prejudice against people of Irish descent.
In the middle 1800s, tens of thousands of refugees, fleeing the Irish potato famine, flooded the eastern ports of the U.S. Those who were fortunate to survive the Atlantic trek in the 'coffin ships' were the poorest of poor. Many of them gave all they had to bring their families to the shores of the U.S. others contracted as indentured servants to wealthy land owners once they made landfall. They were poor, unskilled farmers and they had the mindset of slaves. And "worst of all", unlike the Scotch-Irish who came to Americas during the colonial time, they were Roman Catholic. Even before the potato famine began, anti-Catholic mobs in the U.S. had torched houses and businesses owned by Roman Catholics. The addition of these Irish immigrants simply added to the hotly-burning racist fire.
Most wealthy, at that time, were Protestant. They owned the businesses, farms, and the newspapers. Therefore, it was easy to foment a psychological campaign against the Irish rabble. All they had to do was to convince the public that they wanted to steal the jobs of "hard-working, God-fearing" people and spread new diseases within the U.S.. Much like the German campaign against the Jews, the Atlantic coast powers that be created all kinds of untruthful, prejudicial ideas to marginalize the Irish immigrants. And, like the Jews could "easily be identified by their big noses", the fiery red hair of the Irish was a tell-tale sign you were being invaded by those pesky mongrels.