Using Computer Vision in Table Tennis

[OvrChkn]

Hey everyone, wanted to share an old weekend project that I worked on last year. I used OpenCV in Python to create landmarks, or points on my body to do an approximate pose estimation. I then included a wireframe just to show potential applications in form analysis. The video of my practice was done in November 2022 (I have to do an update eventually using @NextLevel's advice!). What's cool is that you can use this for any type of video where the subject is clearly visible, so I added a clip of FZD.

Let me know what you think!

 

[gordonluvsu]

That looks reaaaallllyy cool.
I could see that being very useful and insightful.
Id like to see more of that.

 

[BaRanchik]

That's awesome. Is that a public project on Git?

 

[OvrChkn]

That's awesome. Is that a public project on Git?

Thank you! I haven't put it on there yet. I wanted to tinker around with it some more, but my computer can't handle some of the advancements I wanted to make. From memory, I used this tutorial:

@gordonluvsu Thank you so much! I love the analytics side of things, so it's cool to see what the potential is for it. I've followed Driveline Baseball, a baseball research lab/company out here in the US, and their biomechanic research and analytics are really top notch.

 

[brokenball]

This is excellent! It needs to be expanded or extended to calculate velocities and accelerations of the points/dots.
This way we can see the acceleration of the paddle hand.. 24 FPS is not very fast. That is only one frame every 40+ ms. There wouldn't be enough frames per TT stroke to make good 3D interpolations. 60 FPS or even120 FPS is much better and can be recorded with phones or iPads. It is possible to interpolate between the points if there enough of them. A high speed camera would be even better.

It is possible to calculate/estimate polynomials for the position, velocity and acceleration for the dots.

So many things to do. I have the equipment to do video processing and take high speed video. I am almost 70 and I have the mental knowledge and energy to want to do these things but lack the physical energy. I need to take naps.

 

[Kuba Hajto]

I wonder if we can use this to allow users of this forum to submit footage without feeling shame or insecurity. If the ball and table was actually captured, that should be enough to make decent estimations.

 

[brokenball]

I wonder if we can use this to allow users of this forum to submit footage without feeling shame or insecurity.

There is a thread for that. It is the video safe thread.

If the ball and table was actually captured, that should be enough to make decent estimations.

What estimations? The answer is yes if you don't expect too much. The calculations are going to be extreme.
More than one camera will be required. They must be synchronized by frame. The frame rate should be about 1000 FPS. This way if a ball is moving 20 m/s the ball will move only 20mm or half a balls width during that time. The spin can be estimated by how it generate the Magnus effect and how the force due to the Magus effect is different from the force of gravity. The estimating the contact will be hard. In my simulations, one degree of error can have a significant affect on the ball's trajectory.

 

[OvrChkn]

I wonder if we can use this to allow users of this forum to submit footage without feeling shame or insecurity. If the ball and table was actually captured, that should be enough to make decent estimations.

I actually tried to capture the ball in a separate video from a top-down perspective. The main issue is my camera, so the ball becomes rather oblong in video and my code couldn't recognize it. I'd have to apply a filter to recognize the unique shape of it and that sent me down a rabbit-hole of looking at computer vision academic papers in sports.

I tried seeing if I could just generate a test-case scenario with getting the coordinates for each landmark on the body into an array, but my computer blue screened and crashed haha

I'm more of an analyst than a computer vision guy, and my job takes up a lot of my time. Wish I could dedicate more time to figuring out a proper performance measurement tool or get a job at a bio-performance research company.

 

[brokenball]

I actually tried to capture the ball in a separate video from a top-down perspective. The main issue is my camera, so the ball becomes rather oblong in video and my code couldn't recognize it. I'd have to apply a filter to recognize the unique shape of it and that sent me down a rabbit-hole of looking at computer vision academic papers in sports.

I tried seeing if I could just generate a test-case scenario with getting the coordinates for each landmark on the body into an array, but my computer blue screened and crashed haha

I'm more of an analyst than a computer vision guy, and my job takes up a lot of my time. Wish I could dedicate more time to figuring out a proper performance measurement tool or get a job at a bio-performance research company.

I have a high speed camera that can capture color videos at 2000 FPS easily but there are limitations and these will apply to what you are trying to do.

The problem is that the shutter can't be one more than a half of the frame period. For instance, at 2000 FPS the period of a frame is 500 microseconds. I have set the frame aperture period to 500 microseconds, the ball will like smeared or elongated as you say. If I set the aperture time to 200 microseconds, I get relatively sharp images BUT then there is yet another problem! Lighting!!!! To only have the aperture on for 200 microseconds required very good lighting and you will not find that under normal conditions. You need to have more light. Professional photographers KNOW how important lighting is. When taking high speed videos the extra lighting is a must.

If you want to keep your videos sharp you need shorter aperture times and than requires more lighting.
This is a video I took over 10 years ago when I only had a black and white high speed camera. The video is taken at 2000 FPS with a 200 microsecond aperture time. The images are sharp but noisy because there isn't enough lighting. I was standing in front of 1500 watts of halogen lamps. I couldn't see the ball because the light was so bright and I couldn't stand in front of the halogen lamps for too long because they were very hot. The halogen lamps will start to melt plastic if left on for too long.

I now have a much better color high speed camera and LED lights. We use this equipment to take high speed videos of machinery that cuts the skin, rot, high sugar and other defects out of french fries.

The solutions are not cheap and they aren't easy.

I think the Omron robot has a good start on playing TT. Apparently it look at the players paddle but doesn't care about what the rest of the player's joints are doing. I think that is the right approach for TT. The Omron robot is mechanically limited, not computer or other wise technically limited. It is the mechanic that are the weak points.

 

[OvrChkn]

I have a high speed camera that can capture color videos at 2000 FPS easily but there are limitations and these will apply to what you are trying to do.

The problem is that the shutter can't be one more than a half of the frame period. For instance, at 2000 FPS the period of a frame is 500 microseconds. I have set the frame aperture period to 500 microseconds, the ball will like smeared or elongated as you say. If I set the aperture time to 200 microseconds, I get relatively sharp images BUT then there is yet another problem! Lighting!!!! To only have the aperture on for 200 microseconds required very good lighting and you will not find that under normal conditions. You need to have more light. Professional photographers KNOW how important lighting is. When taking high speed videos the extra lighting is a must.

If you want to keep your videos sharp you need shorter aperture times and than requires more lighting.
This is a video I took over 10 years ago when I only had a black and white high speed camera. The video is taken at 2000 FPS with a 200 microsecond aperture time. The images are sharp but noisy because there isn't enough lighting. I was standing in front of 1500 watts of halogen lamps. I couldn't see the ball because the light was so bright and I couldn't stand in front of the halogen lamps for too long because they were very hot. The halogen lamps will start to melt plastic if left on for too long.

I now have a much better color high speed camera and LED lights. We use this equipment to take high speed videos of machinery that cuts the skin, rot, high sugar and other defects out of french fries.

The solutions are not cheap and they aren't easy.

I think the Omron robot has a good start on playing TT. Apparently it look at the players paddle but doesn't care about what the rest of the player's joints are doing. I think that is the right approach for TT. The Omron robot is mechanically limited, not computer or other wise technically limited. It is the mechanic that are the weak points.

That makes sense, thank you for the explanation! Cost is definitely a factor unfortunately. I wonder if that can be overcome in one of the filters I was talking about or some sort of convolutional neural network that can handle blob detection. Cause I've seen the ball tracking being done in a recorded tennis environment at such a low FPS. I think they have their code on there too.

I know of the SAS Lab in Raleigh that has a Batting Lab where they break down your swing and attempt to optimize it. Granted this is just stationary, but it'd be so cool to see this for TT.

 

[brokenball]

That makes sense, thank you for the explanation! Cost is definitely a factor unfortunately. I wonder if that can be overcome in one of the filters I was talking about or some sort of convolutional neural network that can handle blob detection. Cause I've seen the ball tracking being done in a recorded tennis environment at such a low FPS. I think they have their code on there too.
The Kalman filter will help but system identification is necessary to determine what the transition matrix should be.

This is my YouTube channel "Peter Ponders Pid".
Most people can't understand the math past 3 minutes.

I know of the SAS Lab in Raleigh that has a Batting Lab where they break down your swing and attempt to optimize it. Granted this is just stationary, but it'd be so cool to see this for TT.

Yes, this just shows how backwards the TT people/forums are. I have the equipment and I have the knowledge but the TT forums have pissed me off so much I don't give a sh!t. I have made my millions so why do I care?

I was going to say if the TT forums would support the engineering students at Washington State University in Vancouver Washington I could support the research but now I am beyond the point of caring.

 

[OvrChkn]

As an update to this, I found a tutorial on a more efficient way to get the poses into a DataFrame format + an interesting way to use Data Science algorithms to label a specific action. So, I spent an hour to get 1 minute worth of pro forehands and 1 minute of backhands towards the building of a classification system. I make sure to get different angles and a decent mix of players, but I used a lot of FZD and Dima to train the data.

I just used a video of Chen Meng's backhand to see if it would properly classify. My big takeaway is that I need to train on more video angles and have a lot more data for a performance increase.

The funny thing is that I ran an old video of me and it classified my forehands as backhands and vice versa.

 

[Tony's Table Tennis]

This is cool
I just want to make a note that training normally will have proper technique
match play, the technique won't be perfect,

so for learning, it is wiser to use training videos than oppose to match videos.
hope that makes sense

 

[palguay]

There is already a model that uses CNN to do a lot of analytics for table tennis. Take a look at this Github implementation of TTNet, it has not been updated but is easy to spin an aws gpu instance and fix the issues to train and start using it rather than doing it from scratch

I have written a post on how to train a model to detect a TT ball. this was a long time ago and it is so much easier now without doing all those things

 

[Trevize84]

A user in the forum suggested the use of AI for setting up drills with a robot. I suggested him to close the loop by letting the AI use a camera to watch the movement and analyze what kind of drills are needed to work on most common mistakes. It seems that your work pretty much covers last mile needed to put this in place.

 

[OvrChkn]

@Tony's Table Tennis Thanks! Definitely agree with you for the training data. The bulk of the data I used to train the model was from different training sessions with like one Truls forehand in a match and a few Hariomoto forehands when he was in a practice match. I think the volume of the data I have + the amount of different angles I have to cover would help improve the performance of the model.

@palguay Thank you for the link. This was a pretty good exercise to start to figure out the individual testing that I'd ultimately want to do. It's a nice starting point to go deeper on the analysis, so I think the next evolution in my testing is to integrate more physical sensors in training.

@Trevize84 That would be a pretty cool integration. There's definitely a lot of model inputs that would have to go into it as well as labeling/identifying key points to capture with personal training. I'm slowly going down the rabbit hole of possibilities haha.

 

[Trevize84]

I'm more into embedded programming so I didn't have a chance to look into AI. However what I would do is to train the AI with thousands of professional matches from top 100 players so that it can understand right movements. Once the AI knows what the right movements are, it can assess how far your movements are from pro-players. Hence propose drills to fill tech gaps and assess how effettive the proposed drills are, so that it learns how to generate more and more effettive customized drills.

 

[brokenball]

So what are you going to do with the data?
BTW, I like here BH. I didn't like her FH. It seemed she didn't have enough time for the FH stroke.

 

[Trevize84]

@OvrChkn like @Tony's Table Tennis said it might also be good idea to train the AI by using videos from pro-players warm ups sessions. Training videos are probably less available, at least with a decent camera angle.

 

[Trevize84]

I have written a post on how to train a model to detect a TT ball.

Unfortunately this link is broken could you post another link?