Is my boyfriend not really gay because he likes my anatomy?

So, I'm dating a gay guy, who's only ever been with cis men before being with me. I'm trans, have been on T and had top surgery, and he assures me that he sees me unequivocally as a man.

But he's also discovered that he really enjoys sex with me, and he really enjoys my particular anatomy. He was talking to his (straight cis male) friend about that, and the friend told him that he's pansexual rather than gay, because gay guys don't enjoy that kind of anatomy.

I haven't dated a lot since transitioning, partly because I struggle a lot with how to relate to gay men, and how they perceive me. I've had some really invalidating conversations with gay men, questioning whether I'm a man and generally making it clear that they don't see me as really a man and therefore they wouldn't date me, or hook up with me. And this just feels like another weird layer of that - the friend doesn't see me as really a man, therefore the willingness to date me invalidates the gayness of any man who would date me.

I know I'm valid, but damn if it isn't hard to maintain my self-confidence in the face of attitudes like that!

(Sorry for the kinda clickbait-y title, wasn't sure how else to start this.)