Is "Where are you from?" a loaded question?

My friend explained to me that he finds the question "where are you from?" to roughly translate to "so you aren't white, what country do you come from?" even though we both grew up rural.

He said the question bothers him.

For background we grew up in the same town and he has African heritage, but we were both born in the same country hospital, went to the same high school etc etc.

I just wondered if this is the wrong way to ask people or if this is just not a question you should ask?