I know in certain circles it is a complete NO NO but i do apply a very thin coat of water based polyurethane on both my blade surfaces AND my handles.
It makes the handle easier to keep clean.
A good way of cleaning the handle might be a clean rag soaked in alcohol based hand -
sanitizer.
Just give it a good rub.
If this will not make it clean enough for you , you might have to take a 400 grit sandpaper to it.
It is after all just a piece of wood . Sand it till its clean enough for you , then use finer sandpaper to make it nice and smooth again.
Here is a picture of my new Hammer, see it is just a piece of wood and i rasp a bit off every day until it fits my paw