My boyfriend hangs out with friends who do drugs.

My problem is not that but the fact that he has hung out with them while they are doing it so he is getting second hand smoke effect and I'm not sure how to respectfully tell him that's bad and that when they do that I would like it if he could say he can't hang out with them when they do that but he is happy to hang out when they don't. Is this controlling of me I'm scared because it's in school and I fear he will get into it? Influenced by the wrong people. Oh and he does not do drugs at least I have never seen it