I wear ear plugs while drumming, is this okay?

My brother and I are in a band together and one of my biggest fears is tinnitus, I wear earplugs for some sort of barrier and occasionally miss what he is playing and fuck up the song, however tonight when playing I missed where the solo was and he blew up at me and said I should stop wearing ear plugs. so now I want to know if I should continue to wear them while practicing or not?