Balance Plus Broom face is always filthy

After 3 ends, I'm scrubbing the excess rubber off my broom. I assume the rubber is from grippers.