Yocto Recipe for Accelerated Video Playback with Qt on Raspberry Pi
Raspberry has a huge community that creates a considerable amount of material in a large number of projects. A relevant support for Yocto cannot therefore be missing for it. Months ago I started to work on using Yocto to create images for the Raspberry Pi. Unfortunately I had to stop for a few months to work on other projects, so the result could not be published immediately: the project is based on Pyro while current Yocto release is Rocko. The concept is however similar.
I created a test image for the Raspberry Pi 3 board, using the best compilation flags I found available, and added a recipe for POT to include hardware acceleration in Qt. The result is good: just invoke qmlscene on a little QML code and you have a hardware accelerated OpenGL scene rendering video with Qt API (QML) in your Yocto image.
You can find the layer here: https://github.com/carlonluca/meta-pot. It is based on the POT project that can be found here: https://github.com/carlonluca/pot.
Images
In the repo I also added a couple of examples of images I created for me. I’ll improve those in case I’ll find the time to do it.
All you have to do is include the recipe and POT will be installed in the image (satisfy the dependencies if some were missing).
Example
I created some images using this setup (I included many things that are not mandatory at all for POT):
BBLAYERS ?= "
${WORKSPACE}/poky-pyro/meta
${WORKSPACE}/poky-pyro/meta-poky
${WORKSPACE}/poky-pyro/meta-openembedded/meta-oe
${WORKSPACE}/poky-pyro/meta-openembedded/meta-multimedia
${WORKSPACE}/poky-pyro/meta-openembedded/meta-networking
${WORKSPACE}/poky-pyro/meta-openembedded/meta-python
${WORKSPACE}/poky-pyro/meta-raspberrypi-crypto
${WORKSPACE}/poky-pyro/meta-qt5
${WORKSPACE}/poky-pyro/meta-oracle-java
${WORKSPACE}/poky-pyro/meta-java
${WORKSPACE}/poky-pyro/meta-office
${WORKSPACE}/meta-pot
"
in my layer.conf I placed:
LICENSE_FLAGS_WHITELIST = "commercial oracle_java"
DISTRO_FEATURES = "ext2 ext3 ext4 pam gles2 usbhost ${DISTRO_FEATURES_LIBC}"
DISTRO_FEATURES_BACKFILL_CONSIDERED += "pulseaudio"
PACKAGE_CLASSES = "package_deb"
IMAGE_FEATURES += "package-management ssh-server-openssh"
DISTRO_FEATURES_remove = "X11 wayland"
EXTRA_IMAGE_FEATURES = "debug-tweaks dev-pkgs tools-sdk tools-debug"
PREFERRED_PROVIDER_jpeg = "libjpeg-turbo"
PREFERRED_PROVIDER_jpeg-native = "libjpeg-turbo-native"
PREFERRED_PROVIDER_udev = "eudev"
PREFERRED_PROVIDER_virtual/java-initial-native = "cacao-initial-native"
PREFERRED_PROVIDER_virtual/java-native = "cacao-native"
PREFERRED_PROVIDER_virtual/javac-native = "ecj-bootstrap-native"
VIRTUAL_RUNTIME_init_manager = "sysvinit"
MACHINE_FEATURES_remove = "apm"
IMAGE_FSTYPES ?= "rpi-sdimg"
MACHINE = "raspberrypi3"
DISTRO = "poky"
At this point you can use something like pot-minimal (https://github.com/carlonluca/meta-pot/blob/master/images/pot-minimal.bb) and you should have a working image.
You can test it pretty simply by using a trivial QML like this:
import QtQuick 2.2
import QtMultimedia 5.5
Rectangle {
color: "orange"
Video {
autoLoad: true
autoPlay: true
source: "file:///home/root/bbb_3m.mov"
anchors.fill: parent
}
}
and you should see your video in the QML scene right away with the command:
qmlscene -platform eglfs main.qml
Optimization
I wanted to also create an image fully optimized for the Raspberry Pi 3 board, which is a Cortex-A53 armv8-a architecture which includes crc and new A64, A32, and T32 instructions to Advanced SIMD that accelerate Advanced Encryption Standard (AES) encryption and decryption, and the Secure Hash Algorithm (SHA) functions SHA-1, SHA-224, and SHA-256. Still I wanted a fully 32-bit system as hardware libraries are only supported for this bitness. To do this I had to apply a few patches to the Yocto repos. I uploaded every patch to these forks:
https://github.com/carlonluca/poky.git (branch: pyro_cortexa53)
https://github.com/carlonluca/meta-raspberrypi (branch: pyro_cortexa53)
Bye! 😉
13 thoughts on “Yocto Recipe for Accelerated Video Playback with Qt on Raspberry Pi”
Leave a Reply
Related Post
Isogeometric Analysis: Knot Insertion for NURBS
Introduction In the classical Finite Element Method (FEM) the domain of the problem is approximated [...]
So it was in fact a broken video that had height not divisible by 2. So that's what " COMXCoreComponent::DecoderEventHandler OMX.broadcom.egl_render – OMX_ErrorInsufficientResources, insufficient resources)" was trying to say 😀 Fixed the source, it plays fine now
Sorry for spamming 😀 I've narrowed the issue to the source video file.
Broken video
Duration: 00:00:04.00, start: 0.000000, bitrate: 333 kb/s
Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709, progressive), 440×247 [SAR 1:1 DAR 440:247], 281 kb/s, 23.98 fps, 23.98 tbr, 24k tbn, 48k tbc (default)
V:PortSettingsChanged: 440×248@23.98 interlace:0 deinterlace:0 anaglyph:0 par:inf display:0 layer:0 alpha:255 aspectMode:0
Video codec omx-h264 width 440 height 247 profile 100 fps 23.976025
Audio codec aac channels 2 samplerate 48000 bitspersample 16
(says 10:54:22.909 ERR: COMXCoreComponent::DecoderEventHandler OMX.broadcom.egl_render – OMX_ErrorInsufficientResources, insufficient resources)
Working video:
Duration: 00:00:12.06, start: 0.000000, bitrate: 1399 kb/s
Stream #0:0(eng): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709), 440×504 [SAR 1:1 DAR 55:63], 1397 kb/s, 23.98 fps, 23.98 tbr, 24k tbn, 50 tbc (default)
Video codec omx-h264 width 440 height 504 profile 100 fps 23.976025
Audio codec aac channels 2 samplerate 48000 bitspersample 16
V:PortSettingsChanged: 440×504@23.98 interlace:0 deinterlace:0 anaglyph:0 par:inf display:0 layer:0 alpha:255 aspectMode:0
(plays fine)
That's with gpu_mem=512 🙂 Can't get any higher. All videos seems to be initialized at once and I think that's too much for POT to handle. Looking on ways to serialize video playback so that resources are allocated for a single video at once
Oh, I guess, nevermind for this issue. I set hdmi_mode to be exactly 1024 by 768 so I probably can't expect to stuff work properly when there's another physical display resolution. It works occasionally, but I assume that doesn't extend to every case.
Of my integration issues, I still seem to hit "COMXCoreComponent::DecoderEventHandler OMX.broadcom.egl_render – OMX_ErrorInsufficientResources, insufficient resources". I have an assortment of QML files with Video inside loaded at once. And it looks like that half of them work and half – don't. Not everything is shown/loaded/played at once, so I hoped this would work 🙂 But it doesn't 🙂
I can see an "10:24:36.224 ERR: COMXCoreTunel::Deestablish – Error WaitForCommand port 220 on component OMX.broadcom.egl_render omx_err(0x80001000)" as well… So the issue seems much like this one https://github.com/carlonluca/pot/issues/60
You mean that if output resolution is 1024×600 you have garbage on the video surfce? Or on the entire screen?
Thank you, Luca!
I've removed all other options from /usr/lib/qt5/plugins/mediaservice/ and it now seems to work 🙂 CPU usage dropped to 2-7% and I can see POT messages when QT_DEBUG_PLUGINS=1 is set. So thank you very much for the hint.
I got another integration question, though 😉 I've used 1024 by 600 display and I got garbage on the screen. It seems to work nice with 1024 by 768, though, so I hope it will work fine with my setup. Any way I can influence its behavior with weird resolutions? Or it is built-in Raspberry-something?
You can see details using the env variable QT_DEBUG_PLUGINS. But POT won't be used if other plugins are available. Qt tends to use other plugins when others are installed. It is typically as simple as moving the other plugins somewhere else.
It depends on you. The code is identical. If you want to build up your own rootfs using yocto you can't use the raspbian build.
Thank you for the great work you done with this project!
It seems like I was able to integrate POT plugin to my Yocto firmware, though I'm unsure if it is actually utilized for decoding h264 video I have here 🙂
Logs say:
“`
qt.multimedia.video: found videonode plugin "egl" 0x16674e8
“`
And I don't know if that's mutually exclusive with POT. It works, but it eats 30% cpu for a simple video.
Is there a way to verify everything is working right?
Where can I find 'meta-raspberrypi-crypto'? Looking on google only this post makes reference to it.
Thanks
I would like to ask, what is the best way (for better performance) to use pot. Is it better to used a yocto image or using Raspbian Strech
After adding qtdeclarative-tools to pot-minimal.bb i can use qmlscene now.
Thank you for making recipe. I tried compile pot-minimal.bb, and it went well. But unfortunately there isn't any qmlscene bin file. I tried to look for it and it looks like it's part of qtdeclarative-tools(oe-pkgdata-util list-pkg-files qtdeclarative-tools) but it wont let me add package using bitbake qtdeclarative-tools. Im really new in Yocto so maybe it's trivial problem.