- Sponsor
-
Notifications
You must be signed in to change notification settings - Fork 56.2k
Add Android Media NDK video I/O file capture back-end #14005
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Tried to rebase onto 3.4 but looks like the Video I/O interfaces have changed quite a bit. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Awesome! Thank you for the contribution!
I believe "master" branch is fine for now.
this->mediaCodec = codec; | ||
this->sawInputEOS = false; | ||
this->sawOutputEOS = false; | ||
AMediaCodec_start(codec); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It make sense to check return value of AMediaCodec_start()
call (at least)
const char *mime; | ||
if (!AMediaFormat_getString(format, AMEDIAFORMAT_KEY_MIME, &mime)) { | ||
LOGV("no mime type"); | ||
return false; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
AMediaFormat_delete(format);
return false;
} | ||
AMediaFormat_delete(format); | ||
} | ||
return true; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if (this->mediaCodec == NULL)
{
if (codec) AMediaCodec_delete(codec);
if (extractor) AMediaExtractor_delete(extractor);
}
return this->mediaCodec != NULL
close(fd); | ||
if (err != AMEDIA_OK) { | ||
LOGV("setDataSource error: %d", err); | ||
return false; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Avoid resources lost:
AMediaExtractor_delete(extractor);
return false;
Consider using of std::shared_ptr
with custom deleter to avoid such manual resource management, like here.
effb803
to
f87afad
Compare
|
@alalek Really appreciate your work on enabling videoio APIs on android. We are very interested in using it. Do you happen to know the timeline for getting this merged into master? |
@yagnasrinath Thanks to @komakai ! Feel free to try this patch and provide feedback. |
Thank you @komakai. I tried out this patch and it seems to work fine for our use cases. Is there anything that we could help you with to get this patch merged into master ASAP? |
@yagnasrinath You're welcome |
@alalek Will this be available in the Android pre-built provided by OpenCV ? Or does it have to be compiled ourselves? |
You can try to download artifacts from "Android pack" builder of this PR. Feature should be available for arm64 / x86_64 architectures. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well done! Thank you 👍
@yagnasrinath not sure what you mean. |
Unless the API LEVEL is set to 21 for that build then probably that wouldn't work. |
@komakai sorry for not being clear, I was talking about support for VideoCapture.open to read from camera device. |
@yagnasrinath You may want to subscribe on this issue: #11952 |
@komakai I tried using your PR. What would be the device path that I need to pass? I gave "1" as the device and I see this error |
@yagnasrinath you need to call the |
@komakai I am not able to open the camera. Here are the errors:
|
@yagnasrinath thanks for testing
Ah - I also got error -30001 (no buffer available) - seems this is not a fatal error just a timing thing. |
@komakai No matter how much ever sleeps I put, I still get read false, after open. I keep getting the same error. Do you think it is better to get the status of repeating request by passing a captureCallback object here https://github.com/komakai/opencv/blob/android-ndk-camera/modules/videoio/src/cap_android_camera.cpp#L367 instead of nullptr? May be that's when we will know that the capture is complete and buffer is available? |
Yeah - I think that's the way to go - will try it out when I get a minute |
@yagnasrinath I've added some synchronization around the |
@komakai Unfortunately, I still see the same error.
|
@yagnasrinath Can you try out the Android NDK camera sample ("basic") at https://github.com/googlesamples/android-ndk/tree/master/camera (just to make sure your device works with Android NDK camera). Also what version of Android are you on? |
I tried the google sample. It seems to works fine. But I could not get the binary built with opencv, working. We are busy with a release. I will try out the code on a different device after the release. |
I tried as much as possible to copy the Google sample code; next step would be to try and work out what my implementation is doing differently. The only thing that comes to mind immediately is that I take the first output resolution available where as the Google sample looks for an output resolution matching the screen resolution |
I have a working c++ opencv program running on windows, and trying to adapt it to run on android. |
Yes
No - this PR uses the Android Media NDK to do the decoding so you don't need ffmpeg - note that Android Media NDK is only available on Android 5 or higher |
@komakai Sorry to bother you.. Can you let me know whether with this patch i can decode MJPG based network streams on android using opencv, or i still need to compile/build with ffmpeg.. Also if you can point me to the right direction for build steps with ffmpeg using NDK 21 for android ? |
@jogiji this patch does not include support for MJPG based network streams
Sorry I have no insight into how to build ffmpeg for Android |
Using the pre-built binaries, I am able to successfully read an mp4 file using the Android Media NDK backend in a 64-bit (arm64-v8a) Android app. However, I need a 32-bit (armeabi-v7a) compatible version and I cannot seem to use I have tried this on numerous devices running Android 10 with identical results (build targets API level 28 and uses NDK 21.3). |
@MichaelBrasco Can you share a stack trace for the crash? (Building with --debug_info should result in meaningful crash dumps). Also - are there any warnings output during the armeabi-v7a build of cap_android_mediandk.cpp (dodgy casts etc.) |
@MichaelBrasco Can you try the following |
@komakai I've encountered 32-bit app crashes under the same conditions as @MichaelBrasco. opencv452_android_VideoCapture_arm64-v8a_success.txt Also, it seems that the same issue was talked about in this thread. I am also looking for a solution to this problem. |
not related to this feature (there is dedicated own MJPEG encoder, Media NDK I/O requires higher Android SDK level which is not enabled in the mentioned distribution). Try to build Android SDK package with debug information. Check logs from original build on public CI. |
The crash in the 32 bit app seems to occur regardless of the type of video file. |
@utibenkei Alternatively you could try to persuade @alalek to change the build configuration for the standard OpenCV Android releases such that the NDK target was set to 21 or higher |
@komakai I'm also using VideoCapture to read and decode video frames from a file in a 32 bit app like @utibenkei , not a camera. I finally managed to stumble upon the fix by specifying the VideoCaptureAPI to CAP_ANDROID. This doesn't seem to cause a difference on arm64-v8a as it's the implicit default, but on armeabi-v7a you will get a crash if you don't provide this argument explicitly. I don't know why this is the case, but it is made even more baffling by the fact that the documentation lists CAP_ANDROID as "not used", which I assume is outdated? There is also a fairly significant performance hit on 32-bit in decoding time, but this is perhaps inevitable. After finding the fix for this fatal issue, another stream of issues ensued. In fairness, I don't think these issues arise from OpenCV's implementation of the Media NDK; my only concern was some of the VideoCapture CAP_PROP_XXX properties were not properly set, so e.g. you cannot seek/rewind videos. On the Samsung S21 we saw a strange green frame flashing every time the video was reset. On some devices, it would just not read the videos at all. We finally had to resort to building OpenCV with FFMPEG for Android. This is a hellish process and is not technically permitted in the CMakeLists, but it is far more reliable and truly cross-platform unlike the Media NDK, where you just have to cross your fingers that the vendor's implementation will be compatible with whatever video format you're using (MP4 AVC is always a safe bet), or it might just not work at all. Here's a guide for anyone wishing to go down this path: https://www.programmersought.com/article/44306233033/ Sorry for hijacking this thread a bit, but this issue has cost me weeks if not an entire month of time on my project, so I hope no one else would have to go through all these steps again. |
@MichaelBrasco |
The colors seem to be swapped when decoding at least mp4 files(android only) wrt. other platforms. |
Current implementation outputs BGR - if you are expecting RGB then yeah, the colors are swapped. |
I don't think this is the problem - we definetely expect BGR and the colors are right on linux, macos, windows and ios. After some heavy digging around i found a pretty detailed discussion exactly on this convoluted badly-documented topic of android formats: Here now we have: The topic suggests: through fourcc.org we can verify this mapping seems to be correct e.g. https://www.fourcc.org/pixel-format/yuv-nv21/ credits for the right search direction: https://answers.opencv.org/question/61628/android-camera2-yuv-to-rgb-conversion-turns-out-green/ |
@kikaxa thanks for the very thorough research. It would be great if you could put together a PR to fix the mapping. |
* Add Android Media NDK video i/o file capture back-end * Fix failing test * Improve error handling/prevent resource leaks * Add license text * Modify default for WITH_ANDROID_MEDIANDK option * Fix spelling of deleter_AMediaExtractor
relates #11952
This pullrequest adds a video I/O file capture back-end for Android