But doesn’t friction NOT depend on contact area and only depend on pressure and smoothness?
NO! It depends on the material involved.
https://www.engineersedge.com/coeffients_of_friction.htm
Do you see area in these equations?
Do you see pressure in these equations?
The idea of pressure isn't completely wrong though. It would equate to friction force per square centimeter or something like that. The problem that a softer sponge will have a MUCH lower impact force for two reasons.
1. The ball that hits the softer sponge has more distance to sink into the sponge before it is stopped so the deceleration forces is weaker.
2. The ball that hits the softer sponge will have its weaker deceleration force spread over a greater area.
The problem is not linear.
The coefficient of friction is multiplied by the force at every point the ball makes contact with the rubber.
This requires some complex math to work out.
Hard rubbers will not let the ball sink into the rubber very far so the deceleration forces are greater than with soft rubbers. Also the harder rubber will not let the ball sink is as far so the contact area is less.
The impact forces might be greater but there is less area. Which would create more friction would require modeling the hard and soft rubbers.