"Semite" is a derivative of "Shemite" or a descendant of Shem, the son of Noah. Shem's descendants include Abram, Isaac (and Ishmael), Jacob, and Judah, from which we get the word "Jew".
Normally, "anti-Semite" is used specifically to describe a person whose words or actions are critical...