First, it's important to read the biblical text!
There is no "curse of Ham" in Genesis 9. Instead, Noah places a curse on Canaan (who in the narrative is Ham's son). And this "Canaan" is obviously the eponymous ancestor of the Canaanites of the land of Canaan, who are the ones the Israelites are subsequently depicted (in the book of Joshua) as depriving of the land, in punishment for their idolatry and wickedness, so the story goes.
The first appearance of a "curse of Ham" is in the Middle Ages, and it's a puzzle where it comes from, as there is no basis in the Genesis text.
Second, the figure Ham in Genesis 9 is the ancestor of the peoples of Canaan, and also Egypt, and perhaps Libya, and Ethiopia. So, by no means simply "black people." Only in the European period of colonialism did the idea appear that the curse of Canaan was transferred to Ham and justified slavery.
source