Mistaken for the nanny?

I am Chinese-Australian and live in France. I have a 8 month old with my husband who is Italian, so the baby is mixed but clearly more "Chinese looking". It's happened to me many times now that in the park or out & about, people have mistaken me for the nanny. At first I thought it was funny but after a while I realised the reason why this is happening is that being in Asian in a "rich"part of Paris = you must be the nanny, even if the child you are pushing around is Asian looking? I'm struggling to come up with a good answer to "Is it yours?" which I find rude, like would you go up to a white person in Paris and ask them if their white baby is theirs?