I never bother to remove glue from the rubber. I do remove it from the blade.
That being said, When gluing I apply glue evenly, but thin, and make sure it’s dried enough. Very porous, light sponges do take a bit more glue, but even so: upon removal, the bulk of it is on the blade and easily...