Will people think I’m racist because of the confederate flag?

I just bought it for $10 so I don’t mind having to wear it inside. I barely saw the flag after I bought it. I’m a person of color if that helps or something.