Your return of his serve looked reasonable to me - maybe could be taken a little earlier and played a little shorter but he wasn't attacking you.
Generally, when you were intentional and confident with your FH you were well on top, but when you backed it off a little then his pushes led to you...