I was figuring I'm just not getting enough spin, but by all logic I should be able to grab onto the ball and spin it just by grazing along it, if you simplify things at least.
In reality, perhaps I do need to sink the ball into the sponge, otherwise it'll slip off even with a tacky rubber.