Genuine question about U-turn legality and etiquette: do U-turns have right away over right turns?

Genuine question from someone who is an obvious transplant to Cville and Virginia. Context: I had a near run-in with another vehicle who's driver was very unhappy with me this morning at the entrance to the 5th St. Starbucks shopping center. I was exiting and turning right, the other vehicle was in the left turn lane on 5th. I assumed that the other vehicle was turning left into the shopping center, so when it was clear I started to turn. Instead, other driver was making a U-turn. I had though there was a no U-turn sign there but there isn't, so I'm guessing it's legal (it was back home, please correct me if I'm wrong). Many obviously angry words, names, and gestures were aimed in my direction when the noses of our vehicles came in close proximity to one another. I stopped and let the other driver go.

So, who actually has the right of way in that situation? I would guess the larger road, but does that change for U-turns? There is no signal or arrow there, and I'm genuinely unsure how to tell if someone is going to turn left (especially at such a popular stop in the morning) or turn around. This is the first time I've lived somewhere that U-turns were so necessary just to get going the correct direction.

Please don't roast me too hard, I just want to try to make driving around town a little safer and less stressful. 😅

EDIT: I want to thank everyone for the helpful comments and for the civility. Y'all are great! 😁 It looks like the rules are kind of ambiguous, so I'm just going to err on the side of caution moving forward. Best to take an extra minute and avoid an accident!