I think "olive skinned" is an expression that is at the polite end of the spectrum of casual racism.
When I was young, I read books that referred to "swarthy Mediterranean types." Anyone described by the author as "swarthy" was likely to be a bad 'un. It was a code that they were not quite white.
You seldom read "swarthy" these days, but "olive skinned" refers to people of the same background: a darker complexion than a white Englishman, and coming from the part of the world associated with olives.
I may be being over sensitive, but I put "olive skinned" in the same column as my Grandma calling Pakistanis "Indians" because she believed it was it was more polite. She had no malice, but was nevertheless making judgements about race and background based on preconceptions.
Preconceptions are low level prejudices.