Both matter, but for the record, the handle scales are primarily designed to match / suit the blade.
If a blade's panel is designed to have flex in it, you can increase or decrease it's propensity to flex by changing the thickness (or the shape!) of the handle scales, through simple increases in mechanical leverage.
Thick handle scales act a lot like a lever... If you make the scales thinner, you shorten the length of the lever, and there's less bending forces applied to the blade during play.
Small changes in thickness don't *usually* result massive changes to a blade's performance in most cases... But with some blades small changes can definitely make a huge difference.
If you like really flex blades in general, I encourage you not to mess with the thickness of their handle scales too much, as it's potentially easy to change the blade's flex, and by extension it's throw angle.
With stiffer blades this is less of a problem, and you have a bit more scope to vary the handle thickness without adjusting the blade's playing feel and performance too much.
That said, really *any* change to a blade's handle geometry has at least some potential to change its playing behaviour to some degree. Predicting how big or how likely any change might be is difficult, as it's going to differ from one blade model to another, depending on its construction.