0

I am creating an android app which will be used to analyze golf shots. I am using cameraX api to get the frames into surfaceview and analyze callback to further processing.

Initially I am using Google Ml-kit object detection to detect the ball but video lags too much that most of the frames are skipped.

How can I use frames in realtime as there maybe 2,3 frames having the ball because the shot will be too fast?

I have tried to gather the frames as bitmaps into an ArrayList and then use that list for further processing but the number of frames vary for a 5 second video.

Should I record the video first for a constant time 5s and then extract frames from it?

Note: The above implementation is for starting because later I need to detect shot in realtime which I tried to detect the ball + detecting the specific sound but in vien.

Furthermore, I need to calculate framrate also which will later be used to calculate the speed of the ball.

Please suggest of any better implementation.

Thanks

PalFS
  • 731
  • 6
  • 16
  • What size image are you using for your detection, the bigger the image the longer it takes which is probably those skipped frames you talk about – tyczj Apr 16 '21 at 19:34
  • I have set the resolution in the image analysis use case as 640X480. – PalFS Apr 16 '21 at 20:54
  • Is there a reason you want to record it as video first then analyze later? If not, ImageAnalysis might be a better choice. The default frame rate is 30fps. Not sure if it's enough to catch the trajectory. Higher fps is also possible but not supported on all devices. – Xi 张熹 Apr 17 '21 at 15:16
  • The reason to record it as video first because I need to analyze multiple frames in order to calculate the speed of the ball. I have tried to use ImageAnalysis but i couldn't find a way to get previous frames. Also, I have no restriction to save a video or show it to the user. Only I have to analyze the frames in order to calculate the speed of the ball and display that speed. – PalFS Apr 18 '21 at 12:42
  • I have some questions about ML Kit Object Detection(ODT) API for your use case. (1) For single image mode, ODT only detects the most prominent center object, but I guess the ball will be only a small part in the image? If you enable multiple objects, it will detect the top 6 prominent objects, but not sure if the ball could be in top 5 and also you need to identify which detected object is the ball. (2) In real time, the ball move very fast, is it still a clear ball shape in each frame? If you break the video into frames, did you verify that ODT can detect the ball in each of those frames? – Steven Apr 20 '21 at 18:49
  • @Steven please find answers for the questions. 1) Yes the ball is a small part but we are using custom object detection with a trained model of a ball. Furthermore, we can move camera closer to the ball so that it can be prominent object in the frame. 2) We have tested this scenario for a 6s video on 30 FPS resulting in 180 frames in which there were only 2 frames having the ball but ODT detected the ball. But unfortunately we have to send those 180 frames to the ODT in order to get the results which another performance issue. – PalFS Apr 21 '21 at 19:22
  • More followups: (1) What performance number are you looking for? Based on the mlkit quickstart app, I can get 30FPS easily on Pixel3XL. Do you need even faster? (2) Do you know how long it take to run with your custom classifier? You could check it by using SINGLE_IMAGE mode with some timer. If that takes long, it could be the major reason for frame skip. (3) Do you need to calculate the ball speed immediately or you can wait couple seconds to get the results? If you can wait, maybe create a queue to store the frames and process them one by one in a separate thread and never drop any frames. – Steven Apr 22 '21 at 21:59
  • 1) There is no performance number, there should be a reasonable amount of time for the analysis. The time can be upto 1min so that results can be displayed to the user for that shot, i.e: speed of the ball. 2) Yes it takes long to detect the image. But I am running the detector in a separate thread asynchroniously so it should not be the reason for the frame skip. 3) We can wait for some time, thanks for the advice we'll try to use queue to store frames. I will update if that helps. – PalFS Apr 24 '21 at 00:05
  • Hi, I have use the queue to store the frames using the STRATEGY_BLOCK_PRODUCER but when I process that queue it raises an exception that image is already closed. Also, ImageAnalysis raises an exception that max image(6) buffer has been aquired, close the image before aquiring more. Is there any way I can make a deep copy of the ImageProxy object? – PalFS Apr 25 '21 at 10:23
  • In the ML Kit quickstart app, there some code for converting media.Image to Bitmap: https://github.com/googlesamples/mlkit/blob/7c711537e266b639c289174307db2b672c1e6414/android/vision-quickstart/app/src/main/java/com/google/mlkit/vision/demo/BitmapUtils.java#L74. If this is too complicated or slow, you may consider trying https://stackoverflow.com/questions/41775968/how-to-convert-android-media-image-to-bitmap-object to only use the first Plane. – Steven Apr 27 '21 at 18:24
  • Thanks for the help. For the first one, I have tried it but some classes were renamed in the newer version of Camera X so there were errors in compiling. Regarding the usage of first plane, this is useful for me as I dont want the image to displayed on screen rather just doing image processing on it. I will try this method and will let you know the results. Thanks – PalFS May 03 '21 at 14:55

0 Answers0