Patience is key. Peel the glue off and apply it again. I hate those DHS sponges; they suck up a lot of the glue and don't spread it very well. But if that's all you have, use a little more glue than normal so you get an even coat.
As for your rubber; is there glue on it? If so, peel the glue and the dirt will come off too. If it isn't glued yet, go ahead and add a couple layers of glue, then peel it off when it dries and you should have a clean sponge. Although I don't think that small amount of dirt will actually affect how it plays if you just want to glue it on your blade.
Maybe choose a cleaner environment to assemble your racket? Or sweep/vacuum the area first...