As was mentioned earlier, the key to a good neutral grip is to be more on the handle and less on the face of the blade. This isn't comfortable for everyone. But I'd you can keep your forefinger consistently parallel to the base of the blade it works. It also makes it easier to create multiple racket angles out of the same grip. The key to racket angle changing is to fold the wrist. Fold it towards your body for backhand, fold it away from your body or leave it neutral for forehand. Though some people like to fold it towards their body for all their strokes.
Grip switching is not bad at all and even in a neutral grip people have mild grip switches. If you want to grip switch, the most important thing is to make them feel natural as part of your backswing and return to ready position and not to make them so slow that you can't do them when neccessary. You probably also need to practice a lot. Recovery from serve is probably the biggest regular grip switch in table tennis. Knowing what grip you end up with after you do that says a lot about your TT.