There is much conflation going on, much of it intentional and sinister, but antisemitism does have a specific meaning & etymology pertaining to Jews, according to Wikipedia:

Due to the root word Semite, the term is prone to being invoked as a misnomer by those who interpret it as referring to racist hatred directed at all "Semitic people" (i.e., those who speak Semitic languages, such as Arabs, Assyrians, and Arameans). This usage is erroneous; the compound word antisemitismus (lit. 'antisemitism') was first used in print in Germany in 1879 as a "scientific-sounding term" for Judenhass (lit. 'Jew-hatred'), and it has since been used to refer to anti-Jewish sentiment alone.