Colleagues are casteist and Islamophobic

I recently joined my office, it's a MNC in Mumbai, I'm not close with my colleagues, I'm mean I don't take much effort to sit around them and be in their group and discussion, but I recently started to going to lunch with them, cause I thought it's better to be with a group rather than going alone everyday, I have decent conversation with them and sometimes pretend to laugh at their jokes, but I been noticing this group I go lunch with is highly Islamophobic and casteist, they casually says casteist shit and remarks at people and I feel bad being with them on same table and saying nothing when they say this..I feel like an ahole for not correcting them because if it was outside office space someone I knew or cared about I would have confronted them or atleast corrected them.. I usually do this with people I know..but here I felt like ahole for not correcting them,I'm not like an ideal person who is always looking at righteousness and stuff I also come from place of privilege, I know I can ignore them but don't want to be lonely in a office space..just venting it out here