# Updates on Hardware Accelerated Qt Multimedia Backend on Raspberry Pi (4.15.2014)

For those of you still working on this exciting subject, I did some new changes to PiOmxTextures and it still works pretty good. To have more information on PiOmxTextures refer to this older article. PiOmxTextures is the code behind the creation of a Qt Multimedia backend to provide hardware accelerated video (and software-decoded audio using ffmpeg) decoding and rendering from/in QML scenes on Raspberry Pi. PiOmxPlayer leverages the code from omxplayer. These are the new additions:

1. Imported code from omxplayer to revision 9b0793faf2c12f49b743ae3778ece03e31ed1538.
2. Now using ffmpeg 2.2.
3. Now it can be built using gcc 4.8 using the recent Raspberry Pi Linaro toolchain released in 2014.
4. Tested using Qt 5.3.0 beta.
5. Tested on the most recent Raspberry Pi firmware: it seems Broadcom has made some changes (fixes?) to the egl_render component. Use the new patch to Qt Multimedia and rebuild. The hack I was forced to add to compensate is no more needed.

Everything seems to keep working correctly, both in 1080p and 720p. Enjoy: https://github.com/carlonluca/pi. Bye! 😉

## 53 thoughts on “Updates on Hardware Accelerated Qt Multimedia Backend on Raspberry Pi (4.15.2014)”

1. Anonymous says:

what should Raspbnerry 3 choose? RP2 or RP1?

2. Anonymous says:

I installed qt5 from repo in archLinuxArm. Can I use PiOmx or should I compile qt5 manually ?

3. What exactly? Tearing is fixed in the no_tearing branch.

4. Hi Luca

I would like to know if you implemented these improvements in the repo sources also?

5. Anonymous says:

Well, now I will compile these additional modules and, if it fails, compilare all over again including these additional modules.

Thank you.

6. I recommend you compile also qttools, qtjsbackend, qtscript, qtxmlpatterns.

7. Anonymous says:

I installed QtQuick1 and QtDeclarative but still keeps asking me the same modules. Will have to recompile QT5 ??

How I can see what are the modules that are installed and running ??

Thank you.

8. QtQuick module is needed. It seems to be missing. Check your Qt installation.

9. Anonymous says:

Hello, I have the following problem. When I want to compile "prepare_openmaxil_backend.sh".

Hello, I have the following problem. When I want to compile "prepare_openmaxil_backend.sh". I've compiled several modules but always keeps asking me the "quick-private and QtQuick" modules.

I use the version qt5.3.2

Someone could help me ?? Thank you.

== ERROR ==

andres@bigwalldev:~/opt/carlonluca-pi/tools$./prepare_openmaxil_backend.sh 4 This script will automatically download and build the dependencies… The openmaxil_backend/3rdparty dir will be cleared. Are you sure you want to go on? Please answer yes or no. The openmaxil_backend/3rdparty dir will be cleared. Are you sure you want to go on? y Preparing 3rdparty dir… Please enter the absolute path to the qmake to use… /home/andres/opt/qt5/qtbase/bin/qmake Ok, about to compile PiOmxTextures… Project ERROR: Unknown module(s) in QT: quick-private qtquick make: *** No targets specified and no makefile found. Stop. make: *** No rule to make target install'. Stop. PiOmxTextures built. Copying libs and headers… cp: cannot stat 'build-PiOmxTextures/libPiOmxTextures*': No such file or directory cp: cannot stat 'build-PiOmxTextures/piomxtextures/*': No such file or directory Cleaning up… 10. Mmm, it's really hard to me to find out where the buffer is stored frame by frame… I was trying to use the given fps as QTimer trigger to avoid the tearing, but I guess I should give up (for a while… not enough time right now). I just want to let you know how I improve some of the issues: – Sometime ago I suggest to change the QSize wide * QSize height * 4 for QSize wide * QSize height * 24 on the texture implementation… I was totally wrong. I was thinking that improve the tearing problem but it was caused by other things. – To improve the weaving effect, I set the DEINTERLACE as AUTO. If I remember well, the OMXVideo.cpp manage to activate it when video size is less than 720*524. This REALLY works. – When I changed between video sources, video and audio get do not sync correctly… specially when it was HD format. I guess this it was a problem generated by av_clock. I solved this by deleting and relaunching the clock and the players each time the source changed. – I remove some stop multimedia events from mediaprocessor and openmaxilplayercontrol. There were, in my opinion, too many unnecessary calls to stop(). Also, I started the cleanup function after the last stop on the mediaprocessor while loop. Moreover,some missing Direct connections were added on the texture generation/invalidation, because without them, there was some kind of bad behavior on the OpenMAXILVideoBuffer creation/destruction because sometimes, after change the media source, it first create the videobuffer and after that it was destroyed. – Finally, I update the OMX lib from popcornmix github. I really hope you manage to keep working on this lib. I am really impressed for how you manage to make OMX lib work side by side with Qt. And of course, I loved! 11. Anonymous says: Well, dont worry. Finally searching through your posts I found what I was missing. I hope to have better luck. Thank you very much. 12. Anonymous says: Hi Luca, I think I skipped a step. These libraries should install directly from an SD memory Raspbian ?? Sorry for this question but for some time now that I was back to 100% compile a version of Raspbian with everything you've managed to give us, and always came across some problems. Now I do not want to leave anything out and stick with it, to create an installation QT5 + video + browser and create a complete guide for all who wish can have a functional system without much knowledge. Also I used a lot to learn more and more about these tools. Well I hope you can guide me a bit. Greetings and thank you very much. 13. Seems clear… ERROR: openssl not found. Do you have openssl in your sysroot? 14. Anonymous says: Hi guys, anyone can tell me why "./compile_ffmpeg.sh" shows me the following error? username@pcdev:~/opt/carlonluca-pi/tools$ ./compile_ffmpeg.sh 4
./compile_ffmpeg.sh: línea 20: /mnt/rasp-pi-rootfs: Es un directorio
Clonar en <>…
remote: Counting objects: 392372, done.
remote: Compressing objects: 100% (85603/85603), done.
remote: Total 392372 (delta 311547), reused 386551 (delta 306110)
Receiving objects: 100% (392372/392372), 94.16 MiB | 1.25 MiB/s, done.
Resolving deltas: 100% (311547/311547), done.
Checking connectivity… hecho.
Note: checking out 'n2.2'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

git checkout -b new_branch_name

HEAD se encuentra en d2c7678… update for 2.2
Configuring…
Prefix to /home/andres/opt/carlonluca-pi/3rdparty/ffmpeg/ffmpeg_src…

If you think configure made a mistake, make sure you are using the latest
version from Git. If the latest version fails, report the problem to the
ffmpeg-user@ffmpeg.org mailing list or IRC #ffmpeg on irc.freenode.net.
Include the log file "config.log" produced by configure as this will help
solving the problem.
Building…
Makefile:2: config.mak: No such file or directory
Makefile:53: /common.mak: No such file or directory
Makefile:94: /libavutil/Makefile: No such file or directory
Makefile:94: /library.mak: No such file or directory
Makefile:96: /doc/Makefile: No such file or directory
Makefile:179: /tests/Makefile: No such file or directory
make: *** No rule to make target /tests/Makefile'. Stop.
Makefile:2: config.mak: No such file or directory
Makefile:53: /common.mak: No such file or directory
Makefile:94: /libavutil/Makefile: No such file or directory
Makefile:94: /library.mak: No such file or directory
Makefile:96: /doc/Makefile: No such file or directory
Makefile:179: /tests/Makefile: No such file or directory
make: *** No rule to make target /tests/Makefile'. Stop.
Cleaning up…
mv: cannot stat 'ffmpeg_compiled/include': No such file or directory
mv: cannot stat 'ffmpeg_compiled/lib': No such file or directory
Done! Bye bye! 😉

Where is the problem ???
Thanks.

15. Anonymous says:

Hi guys, I have to try compile ffmpeg but linux show me the next message:

andres@bigwalldev:~/opt/carlonluca-pi/tools\$ ./compile_ffmpeg.sh 4
./compile_ffmpeg.sh: línea 20: /mnt/rasp-pi-rootfs: Es un directorio
fatal: destination path 'ffmpeg_src' already exists and is not an empty directory.
HEAD se encuentra en d2c7678… update for 2.2
Configuring…
Prefix to /home/andres/opt/carlonluca-pi/3rdparty/ffmpeg/ffmpeg_src…

If you think configure made a mistake, make sure you are using the latest
version from Git. If the latest version fails, report the problem to the
ffmpeg-user@ffmpeg.org mailing list or IRC #ffmpeg on irc.freenode.net.
Include the log file "config.log" produced by configure as this will help
solving the problem.
Building…
mkdir: cannot create directory '/home/andres/opt/carlonluca-pi/3rdparty/ffmpeg/ffmpeg_src/ffmpeg_compiled': File exists
Makefile:2: config.mak: No such file or directory
Makefile:53: /common.mak: No such file or directory
Makefile:94: /libavutil/Makefile: No such file or directory
Makefile:94: /library.mak: No such file or directory
Makefile:96: /doc/Makefile: No such file or directory
Makefile:179: /tests/Makefile: No such file or directory
make: *** No rule to make target /tests/Makefile'. Stop.
Makefile:2: config.mak: No such file or directory
Makefile:53: /common.mak: No such file or directory
Makefile:94: /libavutil/Makefile: No such file or directory
Makefile:94: /library.mak: No such file or directory
Makefile:96: /doc/Makefile: No such file or directory
Makefile:179: /tests/Makefile: No such file or directory
make: *** No rule to make target /tests/Makefile'. Stop.
Cleaning up…
mv: cannot stat 'ffmpeg_compiled/include': No such file or directory
mv: cannot stat 'ffmpeg_compiled/lib': No such file or directory
Done! Bye bye! 😉

Thanks.

16. Hi Luca,
I have "solved" some of my initial problems with the live tv over your library but the tearing problem, so if you don't care, I would like to know what do you think: I guess the problem is that the function onUpdateTriggered() on the videoRendererControl implementation present a frame to the surface when the buffer has been starting changing, so the way to resolve that could be some kind of trigger between the frame and the surface instead the current system based on a QTimer.
What do you thing? Am I totally wrong? Do you thing I there is another approach?
As always, thank you!

17. Hello, I don't precisely know what you're referring to with "quality" and "jitter", but if with vsync you're referring to the horizontal tearing on the video surface I gave some answers on the blog already about that: I never bothered to investigate much but common cause is concurrency. I remember I implemented a patch for that, but I never completed it and I therefore never tested it either.

18. Hey Luca. Do you know why the quality differs between omxplayer and the QT implementation? Is it QT that "breaks" it? There is a noticeable frame jitter and lack of VSYNC on the video. Any idea? (sorry it this is a duplicated comment as i think google didn't post the previous one)

19. Christiano N says:

Hi guys,

I am running into the following debugger problem. Has anyone an idea what might be wrong?

{
"IID": "org.qt-project.qt.mediaserviceproviderfactory/5.0",
"Keys": [
"audiocapture"
],
"Services": [
"org.qt-project.qt.audiosource"
]
},
"className": "AudioCaptureServicePlugin",
"debug": false,
"version": 328450
}

Got keys from plugin meta data ("audiocapture")
defaultServiceProvider::requestService(): no service found for – "org.qt-project.qt.mediaplayer"

20. I'm sorry, I don't have one available anymore. I you have a look at previous articles you may find an old image you can start with.

21. Could you please share your raspberry pi image??? I need this for my university and i am getting crazy!!!

22. B. says:

Hey Luca,

Just finished getting this built with Qt 5.3.1 on the January 7, 2014 Raspbian image. I hit most of the same problems the other commenters have encountered, and your advice to them got me through it as well. Works great.

Thanks so much for sharing this – you've really done some amazing things here.

23. Thanks,Luca and Rİo
this morning ı solve the problem. your tips very helpfull for me..:)

24. @ayberk catar: as said you seem to be building with a c++ library which is incompatible with the one found on the device by the dynamic linker. Just provide the correct one so that the linker uses the one linked during build time. Take it from the toolchain and place it on the device. Use LD_LIBRARY_PATH or any other available technique to make the dyn linker use it.

25. This means the glibc version that you compiled with is newer than the version that is installed and used on the pi.
I solved this by using Arch Linux on my pi. Or you can try to compile with a different tool chain (if I understand this cross compiling correctly) that has the same glibc as the distro you are using on the pi.

26. ı use the plugin debugs and result is :
when ı compile qt5 and carlon pi ı use gcc 4.8 on ubuntu 12.04
ı think have problem with the linkers. but ı am not familiar. can you say how was fix linker problem.
or maybe ı am wrong. can you say waht ı have to do?
thanks..

27. You should use the plugins debug function as Luca Carlon mentioned above.

28. Hİ guys,
First of all thanks for the tutorials.
my problem is same :defaultServiceProvider::requestService(): no service found for – "org.qt-project.qt.mediaplayer"
How can ı solve that. ı have to recompile all qt5, or can ı handle without recompile.
ı compiled ffmpeg with and without –disable-symver –disable-openssl. but result is same.
thanks

29. – Yes, you need the LightLogger project.
– No need to disable ssl. Just provide all the dependencies to ffmpeg. Do you have both libssl and libcrypto in your sysroot?
– You can also simply use the LD_LIBRARY_PATH env variable for quick tests.

30. Hi guys!
First of all, thanks to Rio for all the useful tips and Luca for the huge work.
Finally I've make it work with ffmpeg 2.1. As Rio said, disabling symver and openssl it works, but obviously, there is some garbage under the carpet. I will stay on 2.1 for now but I will try to figure out what is happening with the newer version.
If anyone find something useful, it will really appreciated if it's shared with us.

Just for summarazing, here they are the hot points with ffmpeg 2.2:

– When compiling ffmpeg, –disable-symver and –disable-openssl have to be added to the script.
– For linking the dependencies, you should put the path on /etc/ld.so.conf instead of /etc/ld.so.conf.d/libc.conf. (I don't know why).
– Just for the record, it works in Ubuntu 12.04 and 14.04 (64 bits).

If I've left something, please comment this entry.

Thanks again!

31. I've had the same issue, my sollution was to simply disable openssl when compiling ffmpeg. I didn't need it.
I know it's not the source of the problem but it got me closer to a working program.

32. Hi again,
To solve the previous problem, I recompiled all the qt5 ensuring that the cross-compile fix was done correctly. After that, appears this error:

"/usr/local/qt5/plugins/mediaservice/libopenmaxilmediaplayer.so" : "Cannot load library /usr/local/qt5/plugins/mediaservice/libopenmaxilmediaplayer.so: (libPiOmxTextures.so.4: cannot open shared object file: No such file or directory)"

It has been solved (or at least it seems to) by including the ffmpeg and PiOmxTextures path in /etc/ld.so.conf instead of /etc/ld.so.conf.d/*.conf. When I execute the POCPlayer example all was ok just until the file seems to be opened:

VERBOSE: Opening file:///home/pi/a.mp4…
./POCPlayer: symbol lookup error: /usr/local/qt5/plugins/mediaservice/PiOmxTextures/lib/libPiOmxTextures.so.4: undefined symbol: SSL_library_init

I have the latest version of libssl-dev, so I don't now what the problem could be.
Thanks!

33. Hi again,
Thanks for your fast reply. I've deleted my previous comment because I realized that I didn't moved the ffmpeg and the PiOmxTextures folders to the mediaservices folder.
Unfortunately, I haven't achieved making it work yet. Now, the debugger returns me this error:

Got keys from plugin meta data ("org.qt-project.qt.mediaplayer")
QLibraryPrivate::loadPlugin failed on "/usr/local/qt5/plugins/mediaservice/libopenmaxilmediaplayer.so" : "Cannot load library /usr/local/qt5/plugins/mediaservice/libopenmaxilmediaplayer.so: (/usr/local/qt5/plugins/mediaservice/PiOmxTextures/lib/libPiOmxTextures.so.4: undefined symbol: _dl_hwcap)"
defaultServiceProvider::requestService(): no service found for – "org.qt-project.qt.mediaplayer

I don't understand what it means with libPiOmxTextures.so.4: undefined symbol: _dl_hwcap.
Do you have any clue?

/usr/local/../plugins/mediaservice/PiOmxTextures/lib
/usr/local/../plugins/mediaservice/ffmpeg/lib

Thanks!

34. This comment has been removed by the author.

35. Add it to the default library paths so that the dynamic linker finds it, as usual.

36. This comment has been removed by the author.

37. Sure, I've read the previous post but I'm missing something. I have long been trying different things to create a media player but there are always things that I can not find. I keep looking between the previous post and hopefully I can build the development environment.

It would be good to do a compilation of the steps to follow in order to realize the construction of this project. It would be very helpful for someone who does not have much knowledge on this topic.

Thank you.

38. Follow this and the previous posts. You should find all you need to build.

39. Hi Carlon, well I try to follow the steps to compile this environment but I'm very lost. You can list the steps to achieve compile everything and make a card?

One of the parts where I lost it, how to get qt5pi folder? I have not been able to find the steps to prepare this directory.

Thank you.

40. Thanks, that is a handy variable to know about. I'm just starting out with Qt and C++ but I'm really liking what I see so far.
I was getting some SSL init function errors on my pi because of a version mismatch between my pi and ubuntu install i guess. Disableing ssl in the ffmpeg compile solved that problem.
After copying the libs to /opt and adding it to the linker path ldconfig was complaining about them not being symlinks. Creating them by hand made that go away as well. And now finally I've got it working! There are still some performance issues but I got some improvements by lowering the bit rate of the video. Now all I have to do is get pyqt5 to compile for this qt build and get smooth fades between videos.

Thanks, and if you have any other tips they'd be very much appreciated 😉

41. Use the QT_DEBUG_PLUGINS env variable to get information about the issue.

42. Thanks Carlon, missed the real error for some reason. By adding –disable-symver to the ffmpeg configure I got it to compile on both Ubuntu 13.10 32bit with 32bit toolchain and Ubuntu 14.04 64bit with the 64bit toolchain. However I couldn't get it to play my test video. Still getting the defaultServiceProvider::requestService(): no service found for – "org.qt-project.qt.mediaplayer" error. Must be something with the linking. ldconfig does pick them up but complains about not really being symlinks. Will check it out again this weekend.

43. Just for completeness here are the steps I've used to get to the error. I'm working on a 64bit Ubuntu 14.04 install and an updated Raspbian wheezy image with the packages installed as recommended by both your post and the qt-project wiki.

Quick list of commands used:
Clone and checkout raspberry pi tools and qt5

git clone https://github.com/carlonluca/pi.git
git clone https://github.com/raspberrypi/tools.git
git clone git://gitorious.org/qt/qt5.git
cd qt5
git checkout v5.3.0-beta1
./init-repository

Build QtBase

cd qtbase
./configure -prefix /usr/local/qt5pi -opensource -confirm-license -make libs -make tools -nomake tests -optimized-qmake -no-pch -device linux-rasp-pi-g++ -device-option CROSS_COMPILE=/home/rio/opt/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/bin/arm-linux-gnueabihf- -opengl es2 -sysroot /home/rio/opt/rootfs
Configure summary
make -j4
sudo make install

Build (in this order) QtDeclarative, QtScript, QtImageformats, QtGraphicaleffects

cd /home/rio/opt/qt5/qt{module}
/usr/local/qt5pi/bin/qmake ./
make -j4
sudo make install

run compile_ffmpeg.sh and prepare_openmaxil_backend.sh

cd /home/rio/opt/carlonluca-pi/
wget https://raw.githubusercontent.com/carlonluca/LightLogger/master/lc_logging.h
cd /home/rio/opt/carlonluca-pi/tools
edit compile_ffmpeg.sh to add /home/rio/opt/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/bin to PATH
RPI_SYSROOT=/home/rio/opt/rootfs ./compile_ffmpeg.sh 4
./prepare_openmaxil_backend.sh 4

44. I've been trying to get Qt5 to do hardware accelerated h264 video through qml but still no luck.
Your solution looks like the cleanest and the fastest. But I'm getting this weird error on the prepare_openmaxil_backend script that has me pinned down. I've confirmed that the compiled Qt5 libraries work with qmlscene on the raspberry pi with 3 images crossfading very smoothly on a spacebar press.

Here is the error I'm having. Hope you can help me out.

/home/rio/opt/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/bin/arm-linux-gnueabihf-g++ -Wl,-rpath-link,/home/rio/opt/rootfs/opt/vc/lib -Wl,-rpath-link,/home/rio/opt/rootfs/usr/lib/arm-linux-gnueabihf -Wl,-rpath-link,/home/rio/opt/rootfs/lib/arm-linux-gnueabihf –sysroot=/home/rio/opt/rootfs -Wl,-O1 -Wl,-rpath,/usr/local/qt5pi/lib -shared -Wl,-soname,libPiOmxTextures.so.4 -o libPiOmxTextures.so.4.2.0 openmaxiltextureloader.o Locker.o Event.o OMXComponent.o omxtunnel.o omx_imageelement.o omx_videosurfaceelement.o omx_camerasurfaceelement.o omx_audioprocessor.o omx_mediaprocessor.o OMX_Core.o omx_mediaprocessorelement.o omx_globals.o omx_textureprovider.o omx_playeraudio.o omx_reader.o omx_omxplayer_logging.o Srt.o Unicode.o SubtitleRenderer.o OMXVideo.o OMXThread.o OMXSubtitleTagSami.o OMXStreamInfo.o OMXReader.o OMXPlayerVideo.o OMXPlayerSubtitles.o OMXPlayerAudio.o OMXOverlayCodecText.o OMXCore.o OMXClock.o OMXAudioCodecOMX.o OMXAudio.o File.o DynamicDll.o BitstreamConverter.o XMemUtils.o RBP.o RegExp.o PCMRemap.o moc_OMXComponent.o moc_omx_imageelement.o moc_omx_videosurfaceelement.o moc_omx_camerasurfaceelement.o moc_omx_audioprocessor.o moc_omx_mediaprocessor.o moc_omx_mediaprocessorelement.o moc_OMXVideo.o moc_OMXPlayerVideo.o -L/home/rio/opt/rootfs/opt/vc/lib -lopenmaxil -lEGL -lbcm_host -lvcos -lrt -lv4l2 /home/rio/opt/carlonluca-pi/3rdparty/ffmpeg/lib/libavformat.a /home/rio/opt/carlonluca-pi/3rdparty/ffmpeg/lib/libavcodec.a /home/rio/opt/carlonluca-pi/3rdparty/ffmpeg/lib/libavutil.a /home/rio/opt/carlonluca-pi/3rdparty/ffmpeg/lib/libswscale.a /home/rio/opt/carlonluca-pi/3rdparty/ffmpeg/lib/libswresample.a -lWFC -lpcre -L/home/rio/opt/rootfs/usr/lib/arm-linux-gnueabihf -lfreetype -L/home/rio/opt/rootfs/usr/local/qt5pi/lib -lQt5Quick -L/usr/local/qt5pi/lib -lQt5OpenGL -lQt5Qml -lQt5Widgets -lQt5Network -lQt5Gui -lQt5Core -lGLESv2 -lpthread
/home/rio/opt/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/bin/../lib/gcc/arm-linux-gnueabihf/4.8.3/../../../../arm-linux-gnueabihf/bin/ld: failed to set dynamic section sizes: Bad value
collect2: error: ld returned 1 exit status
make: *** [libPiOmxTextures.so.4.2.0] Error 1

45. Awesome, I look forward to giving it a try, I'm just getting Qt cross-compiled so we'll see how it goes. Fantastic work you've done.

46. Both are available in PiOmxTextures but the technology used is exactly the same, so I don't expect much difference.

47. For a pi based media player that needs to overlay seek bars and other UI elements would you recommend using your QtMultimedia support or using the custom component ?