Of course, if you for example only do bicep curls, and build up large biceps, but you have a very imbalanced arm/back/chest/shoulder area, then that is not good in general and will hamper your table tennis. I don't think we are talking about that.
Just by playing and swinging you will build up...