Gozo, if you count the points you lost, the vast majority of them is not due to the pips but due to his serve and you misreading his serve. He has a more advanced serve than yours - he uses the body decently + he can do both hook serves + pendulum. And he has the classic elbow lift pendulum serve which is really effective in disguising the backspin on the ball (it is basically heavy sidebackspin, not sure why you insisted on treating it like sidespin) .
Obviously he relies on the pips for receive, but tbh your serves aren't remotely as threatening as his, if there was a double inverted player with those serves the effect would have been similar.
Advanced servers do have a way to make you feel like you can't even play your game properly. Unfortunately this is also the hardest part of the game, and where pips/anti would really help you with.