Yes you are on the right track about the weight distribution.
It’s heavy rubber and/or handles that are too thin or hollowed out like Stiga WRB, Donic Senso, etc.
You can try an experiment: sticky-tape a gold coin or fishing weight to the base of your handle. Although heavier, this will shift...