Upon discovering that the word tangerine means ‘inhabitant of Tangier (Tanger in the Romance langauges, Ṭanjah /t̪ʕandʒa/ in Arabic)’, and that the term was used to refer to Sub-Saharan soldiers and slaves in Al Andalus as well as the fruit, it occured to me that the term might be racist, similar to the name nigger toes for Brazil nuts. (Sorry to type that out! But I’m just doing linguistics here.)
Luckily, that’s not the case. It turns out they are native to the area around Tangier, Morocco. (And it was originally an adjective, as tangerine orange.) So they are called ‘of Tangier’ because they were actually imported from there, not because of any racist connection between some appearance of the fruit and a misguided unappreciation of human diversity.