Skip to content

Add Android Media NDK video I/O file capture back-end #14005

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Mar 19, 2019

Conversation

komakai
Copy link
Contributor

@komakai komakai commented Mar 8, 2019

relates #11952

This pullrequest adds a video I/O file capture back-end for Android

  • add necessary CMake config to determine if the Android Media NDK is available (platform must be Android and the API level must be 21 or over)
  • add code for decoding video file frames using the Android Media NDK
  • add the back-end the videoio registry
  • add a config file which specifies API level 21
android_pack_config=ndk-18-api-level-21.config.py

komakai and others added 2 commits March 8, 2019 09:53

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
@komakai
Copy link
Contributor Author

komakai commented Mar 8, 2019

Tried to rebase onto 3.4 but looks like the Video I/O interfaces have changed quite a bit.
Not sure about testing - to access the Media NDK APIs the tests would need to run on an Android emulator.

Copy link
Member

@alalek alalek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome! Thank you for the contribution!

I believe "master" branch is fine for now.

this->mediaCodec = codec;
this->sawInputEOS = false;
this->sawOutputEOS = false;
AMediaCodec_start(codec);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It make sense to check return value of AMediaCodec_start() call (at least)

const char *mime;
if (!AMediaFormat_getString(format, AMEDIAFORMAT_KEY_MIME, &mime)) {
LOGV("no mime type");
return false;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AMediaFormat_delete(format);
return false;

}
AMediaFormat_delete(format);
}
return true;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if (this->mediaCodec == NULL)
{
    if (codec) AMediaCodec_delete(codec);
    if (extractor) AMediaExtractor_delete(extractor);
}
return this->mediaCodec != NULL

close(fd);
if (err != AMEDIA_OK) {
LOGV("setDataSource error: %d", err);
return false;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Avoid resources lost:

AMediaExtractor_delete(extractor);
return false;

Consider using of std::shared_ptr with custom deleter to avoid such manual resource management, like here.

@komakai komakai force-pushed the android-video-cap branch from effb803 to f87afad Compare March 11, 2019 11:16
@komakai
Copy link
Contributor Author

komakai commented Mar 11, 2019

  • switched to use std::shared_ptr with deleters where appropriate and to use std::vector<uint8_t> instead of raw uint8_t* buffer.
  • additional return value checks for API calls
  • modified logic to take the first video track for which decoding could be successfully started

@yagnasrinath
Copy link

yagnasrinath commented Mar 19, 2019

@alalek Really appreciate your work on enabling videoio APIs on android. We are very interested in using it. Do you happen to know the timeline for getting this merged into master?

@alalek
Copy link
Member

alalek commented Mar 19, 2019

@yagnasrinath Thanks to @komakai !

Feel free to try this patch and provide feedback.

@yagnasrinath
Copy link

@yagnasrinath Thanks to @komakai !

Feel free to try this patch and provide feedback.

Thank you @komakai. I tried out this patch and it seems to work fine for our use cases. Is there anything that we could help you with to get this patch merged into master ASAP?

@komakai
Copy link
Contributor Author

komakai commented Mar 19, 2019

@yagnasrinath You're welcome
I think with the small spelling fix we should be really to go.
(At least for an initial version - I will put some notes on #11952 about possible enhancements)

@PhilLab
Copy link
Contributor

PhilLab commented Mar 19, 2019

@alalek Will this be available in the Android pre-built provided by OpenCV ? Or does it have to be compiled ourselves?

@alalek
Copy link
Member

alalek commented Mar 19, 2019

You can try to download artifacts from "Android pack" builder of this PR. Feature should be available for arm64 / x86_64 architectures.

Copy link
Member

@alalek alalek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well done! Thank you 👍

@alalek alalek merged commit 0d1a0b1 into opencv:master Mar 19, 2019
@yagnasrinath
Copy link

@komakai @alalek Is there any chance that open() API is supported for android as well?

@komakai
Copy link
Contributor Author

komakai commented Mar 19, 2019

@yagnasrinath not sure what you mean. VideoCapture.open should be working for files. What are you trying to open?

@komakai
Copy link
Contributor Author

komakai commented Mar 19, 2019

You can try to download artifacts from "Android pack" builder of this PR

Unless the API LEVEL is set to 21 for that build then probably that wouldn't work.

@yagnasrinath
Copy link

@komakai sorry for not being clear, I was talking about support for VideoCapture.open to read from camera device.

@alalek
Copy link
Member

alalek commented Mar 20, 2019

@yagnasrinath You may want to subscribe on this issue: #11952

@komakai
Copy link
Contributor Author

komakai commented Mar 24, 2019

@yagnasrinath
Copy link

@komakai I tried using your PR. What would be the device path that I need to pass? I gave "1" as the device and I see this error NativeCodec: failed to stat file: 1 (No such file or directory)

@komakai
Copy link
Contributor Author

komakai commented Mar 25, 2019

@yagnasrinath you need to call the open(int index) overload i.e. pass an int instead of a String. Also make sure to build with native API level 24

@yagnasrinath
Copy link

@komakai I am not able to open the camera. Here are the errors:

03-25 00:23:51.895 23240 23248 W OpenCV/4.1.0-pre: [ WARN:0] VIDEOIO(ANDROID_NATIVE): trying capture cameraNum=0 ...
03-25 00:23:51.896   179   179 I Camera2ClientBase: Closed Camera 0. Client was: com.foghorn.edge (PID 23240, UID 10061)
03-25 00:23:51.897 23240 23248 W ACameraDevice: CameraDevice: bad count 0 for shading map size
03-25 00:23:51.897   179   179 I CameraService: CameraService::connect call (PID -1 "", camera ID 0) for HAL version default and Camera API version 2
03-25 00:23:51.898   179   179 I Camera2ClientBase: Camera 0: Opened. Client:  (PID 23240, UID 10061)
03-25 00:23:51.898   179   179 I CameraDeviceClient: CameraDeviceClient 0: Opened
03-25 00:23:51.900   179   179 I Camera  : openDevice:0: Opening camera device
03-25 00:23:51.903 23240 23248 E NdkImageReader: AImageReader_getWindow

03-25 00:23:51.906   179   179 D Camera3-Device: Set real time priority for request queue thread (tid 24263)
03-25 00:23:51.907 23240 24259 W NativeCamera: session 0xaf733848 active
03-25 00:23:51.907 23240 23248 W OpenCV/4.1.0-pre: [ WARN:0] VIDEOIO(ANDROID_NATIVE): created, isOpened=1
03-25 00:23:51.907 23240 23248 E NativeCamera: Acquire image failed with error code: -30001
03-25 00:23:51.907 23240 23241 E av-agent: [opencv_player.cc:398] Unexpected halt of live camera "0" because OpenCV's VideoCapture::read returned false.
03-25 00:23:51.907 23240 23241 E av-agent: [av_agent-main.cc:149] Error Handler Called: [Unexpected halt of live camera "0" because OpenCV's VideoCapture::read returned false.]

@komakai
Copy link
Contributor Author

komakai commented Mar 25, 2019

@yagnasrinath thanks for testing

NativeCamera: Acquire image failed with error code: -30001

Ah - I also got error -30001 (no buffer available) - seems this is not a fatal error just a timing thing.
You need to wait until a buffer is available.
As a work-around I put in some sleeps (~500ms) between the call to open and the first call to read
and between each subsequent call to read.
A more robust solution would be to try and find some callback from the NDK indicating when a buffer was available and wait for that before trying to acquire the image.

@yagnasrinath
Copy link

@komakai No matter how much ever sleeps I put, I still get read false, after open. I keep getting the same error. Do you think it is better to get the status of repeating request by passing a captureCallback object here https://github.com/komakai/opencv/blob/android-ndk-camera/modules/videoio/src/cap_android_camera.cpp#L367 instead of nullptr? May be that's when we will know that the capture is complete and buffer is available?

@komakai
Copy link
Contributor Author

komakai commented Mar 26, 2019

@yagnasrinath

Do you think it is better to get the status of repeating request by passing a captureCallback object here https://github.com/komakai/opencv/blob/android-ndk-camera/modules/videoio/src/cap_android_camera.cpp#L367 instead of nullptr?
May be that's when we will know that the capture is complete and buffer is available?

Yeah - I think that's the way to go - will try it out when I get a minute

@komakai
Copy link
Contributor Author

komakai commented Mar 26, 2019

@yagnasrinath I've added some synchronization around the onCaptureCompleted callback and it seems to work without any sleeps - please try it out

@yagnasrinath
Copy link

yagnasrinath commented Mar 27, 2019

@komakai Unfortunately, I still see the same error.

03-27 12:53:28.904 179 4087 I Camera2ClientBase: Camera 0: Opened. Client: (PID 13249, UID 10061)
03-27 12:53:28.904 179 4087 I CameraDeviceClient: CameraDeviceClient 0: Opened
03-27 12:53:28.906 179 4087 I Camera : openDevice:0: Opening camera device
03-27 12:53:28.908 13249 13257 E NdkImageReader: AImageReader_getWindow
03-27 12:53:28.911 179 4087 D Camera3-Device: Set real time priority for request queue thread (tid 13272)
03-27 12:53:28.912 13249 13268 W NativeCamera: session 0xa3db30d8 active
03-27 12:53:28.912 13249 13257 W NativeCamera: No Buffer Available error occured - waiting for callback
03-27 12:53:30.752 13249 13250 I av-agent: [opencv_player.cc:322] In the past 2.0s we've read 0 frames from "0" and emitted 0 of them to topic "/raw/av" (input 0.0 FPS, output 0.0 FPS)
03-27 12:53:30.913 13249 13257 E NativeCamera: Capture failed or callback timed out
03-27 12:53:30.913 13249 13250 E av-agent: [opencv_player.cc:397] Unexpected halt of live camera "0" because OpenCV's VideoCapture::read returned false.

@komakai
Copy link
Contributor Author

komakai commented Mar 28, 2019

@yagnasrinath Can you try out the Android NDK camera sample ("basic") at https://github.com/googlesamples/android-ndk/tree/master/camera (just to make sure your device works with Android NDK camera). Also what version of Android are you on?

@yagnasrinath
Copy link

I tried the google sample. It seems to works fine. But I could not get the binary built with opencv, working. We are busy with a release. I will try out the code on a different device after the release.

@komakai
Copy link
Contributor Author

komakai commented Apr 1, 2019

I tried the google sample. It seems to works fine

I tried as much as possible to copy the Google sample code; next step would be to try and work out what my implementation is doing differently. The only thing that comes to mind immediately is that I take the first output resolution available where as the Google sample looks for an output resolution matching the screen resolution

@HaimBendanan
Copy link

HaimBendanan commented Oct 8, 2020

I have a working c++ opencv program running on windows, and trying to adapt it to run on android.
I am using the android SDK, version 3.4.11 and it seems that reading from video files (cv::VideoCapture(videoPath)) doesn't work (the video is an mp4)
Does this PR fixed that issue? I read on other threads that recompiling opencv with ffmpeg is needed in order to use cv::VideoCapture(videoPath) on Android - is that still the case?

@komakai
Copy link
Contributor Author

komakai commented Oct 9, 2020

Does this PR fix that issue?

Yes

recompiling opencv with ffmpeg is needed in order to use cv::VideoCapture(videoPath) on Android - is that still the case?

No - this PR uses the Android Media NDK to do the decoding so you don't need ffmpeg - note that Android Media NDK is only available on Android 5 or higher

@jogiji
Copy link

jogiji commented Oct 11, 2020

@komakai Sorry to bother you.. Can you let me know whether with this patch i can decode MJPG based network streams on android using opencv, or i still need to compile/build with ffmpeg.. Also if you can point me to the right direction for build steps with ffmpeg using NDK 21 for android ?

@komakai
Copy link
Contributor Author

komakai commented Oct 11, 2020

@jogiji this patch does not include support for MJPG based network streams

Also if you can point me to the right direction for build steps with ffmpeg using NDK 21 for android ?

Sorry I have no insight into how to build ffmpeg for Android

@MichaelBrasco
Copy link

MichaelBrasco commented Feb 5, 2021

Using the pre-built binaries, I am able to successfully read an mp4 file using the Android Media NDK backend in a 64-bit (arm64-v8a) Android app. However, I need a 32-bit (armeabi-v7a) compatible version and I cannot seem to use VideoCapture normally (i.e. with cv::CAP_ANY) without getting a SIGBUS crash. The only backend that I find works with the 32-bit version is cv::CAP_IMAGES with a sequence of images, but this suffers from a significant performance loss, so it's not feasible in my case. Could you please let me know if there is a reason for this problem or any solutions available?

I have tried this on numerous devices running Android 10 with identical results (build targets API level 28 and uses NDK 21.3).

@komakai
Copy link
Contributor Author

komakai commented Feb 5, 2021

@MichaelBrasco Can you share a stack trace for the crash? (Building with --debug_info should result in meaningful crash dumps). Also - are there any warnings output during the armeabi-v7a build of cap_android_mediandk.cpp (dodgy casts etc.)

@komakai
Copy link
Contributor Author

komakai commented Feb 6, 2021

@MichaelBrasco Can you try the following
objdump -t android_build/OpenCV-android-sdk/sdk/native/libs/armeabi-v7a/libopencv_java4.so | grep AndroidMediaNdk
and compare with:
objdump -t android_build/OpenCV-android-sdk/sdk/native/libs/arm64-v8a/libopencv_java4.so | grep AndroidMediaNdk
If the output is different then it is possible that armeabi-v7a build isn't being built with the correct ANDROID_NATIVE_API_LEVEL

@utibenkei
Copy link
Contributor

@komakai I've encountered 32-bit app crashes under the same conditions as @MichaelBrasco.
I downloaded the pre-built version 4.5.2 binary from OpenCV official, created a project in AndroidStudio, and created a test android app that loads MJPEG video files in the VideoCapture class.
It all works fine in the 64-bit (arm64-v8a) build, but in the 32-bit (armeabi-v7a) build, the app crashes.
I have attached the stack trace logs for both 64- bit and 32-bit.

opencv452_android_VideoCapture_arm64-v8a_success.txt
opencv452_android_VideoCapture_armeabi-v7a_crash.txt

Also, it seems that the same issue was talked about in this thread.
#11995 (comment)

I am also looking for a solution to this problem.

@alalek
Copy link
Member

alalek commented Apr 20, 2021

that loads MJPEG video files in the VideoCapture class

not related to this feature (there is dedicated own MJPEG encoder, Media NDK I/O requires higher Android SDK level which is not enabled in the mentioned distribution).

Try to build Android SDK package with debug information. Check logs from original build on public CI.

@utibenkei
Copy link
Contributor

The crash in the 32 bit app seems to occur regardless of the type of video file.
It may not have been appropriate to post this topic in this thread, but it seems that VideoCapture class does not work with 32-bit apps.

@komakai
Copy link
Contributor Author

komakai commented Apr 21, 2021

@utibenkei
The prebuilt binaries target an NDK version lower than the NDK version required for NDK media support.
(Because Android's arm64-v8 support was added quite late the arm64-v8 build ignores NDK versions that are too low and automatically chooses a higher NDK version that it turns out does support NDK media.)
To get a build that will work on armeabi-v7a and arm64-v8 please try building with the following command line:
./opencv/platforms/android/build_sdk.py --use_android_buildtools --no_samples_build --config ./opencv/platforms/android/ndk-18-api-level-21.config.py opencv_android_ndk21 opencv

Alternatively you could try to persuade @alalek to change the build configuration for the standard OpenCV Android releases such that the NDK target was set to 21 or higher

@MichaelBrasco
Copy link

MichaelBrasco commented Apr 21, 2021

@komakai
Apologies for the late reply. My logcat output didn't show a particularly helpful stack trace and the object dumps of the armeabi-v7a binaries have references to the Android Media NDK just like the arm64-v8a binaries.

I'm also using VideoCapture to read and decode video frames from a file in a 32 bit app like @utibenkei , not a camera. I finally managed to stumble upon the fix by specifying the VideoCaptureAPI to CAP_ANDROID. This doesn't seem to cause a difference on arm64-v8a as it's the implicit default, but on armeabi-v7a you will get a crash if you don't provide this argument explicitly. I don't know why this is the case, but it is made even more baffling by the fact that the documentation lists CAP_ANDROID as "not used", which I assume is outdated? There is also a fairly significant performance hit on 32-bit in decoding time, but this is perhaps inevitable.

After finding the fix for this fatal issue, another stream of issues ensued. In fairness, I don't think these issues arise from OpenCV's implementation of the Media NDK; my only concern was some of the VideoCapture CAP_PROP_XXX properties were not properly set, so e.g. you cannot seek/rewind videos. On the Samsung S21 we saw a strange green frame flashing every time the video was reset. On some devices, it would just not read the videos at all. We finally had to resort to building OpenCV with FFMPEG for Android. This is a hellish process and is not technically permitted in the CMakeLists, but it is far more reliable and truly cross-platform unlike the Media NDK, where you just have to cross your fingers that the vendor's implementation will be compatible with whatever video format you're using (MP4 AVC is always a safe bet), or it might just not work at all. Here's a guide for anyone wishing to go down this path: https://www.programmersought.com/article/44306233033/

Sorry for hijacking this thread a bit, but this issue has cost me weeks if not an entire month of time on my project, so I hope no one else would have to go through all these steps again.

@komakai
Copy link
Contributor Author

komakai commented Apr 21, 2021

@MichaelBrasco
Thanks for the write-up. It would be great if you could prepare a Pull-Request that addresses any of (or all of!) the troubles you encountered in your project.

@kikaxa
Copy link
Contributor

kikaxa commented Jul 2, 2021

The colors seem to be swapped when decoding at least mp4 files(android only) wrt. other platforms.
As far as i know this stems from the historical swap of the U/V planes.

@komakai
Copy link
Contributor Author

komakai commented Jul 3, 2021

The colors seem to be swapped when decoding at least mp4 files(android only) wrt. other platforms.

Current implementation outputs BGR - if you are expecting RGB then yeah, the colors are swapped.
I think the correct thing to do is implement the CAP_PROP_FOURCC property and then people can just choose the output format they want

@kikaxa
Copy link
Contributor

kikaxa commented Jul 3, 2021

I don't think this is the problem - we definetely expect BGR and the colors are right on linux, macos, windows and ios.

After some heavy digging around i found a pretty detailed discussion exactly on this convoluted badly-documented topic of android formats:
https://community.khronos.org/t/omx-color-formatyuv420semiplanar-yuv420planar-fourcc-fmts/3807/5

Here now we have:
COLOR_FormatYUV420SemiPlanar -> cv::COLOR_YUV2BGR_NV21
OLOR_FormatYUV420Planar -> cv::COLOR_YUV2BGR_YV12

The topic suggests:
OMX_COLOR_FormatYUV420SemiPlanar — NV12
OMX_COLOR_FormatYUV420Planar — IYUV or I420
OMX_COLOR_FormatYVU420SemiPlanar — NV21
OMX_COLOR_FormatYVU420Planar — YV12

through fourcc.org we can verify this mapping seems to be correct e.g. https://www.fourcc.org/pixel-format/yuv-nv21/

credits for the right search direction: https://answers.opencv.org/question/61628/android-camera2-yuv-to-rgb-conversion-turns-out-green/

@komakai
Copy link
Contributor Author

komakai commented Jul 4, 2021

@kikaxa thanks for the very thorough research. It would be great if you could put together a PR to fix the mapping.

a-sajjad72 pushed a commit to a-sajjad72/opencv that referenced this pull request Mar 30, 2023
* Add Android Media NDK video i/o file capture back-end

* Fix failing test

* Improve error handling/prevent resource leaks

* Add license text

* Modify default for WITH_ANDROID_MEDIANDK option

* Fix spelling of deleter_AMediaExtractor
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
category: videoio effort: few weeks Contribution / porting of a new/existed algorithm. With samples / tests / docs / tutorials feature platform: android
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

9 participants