Render yuv420p Frames in OpenGL with GPU-accelerated conversion

Rendering YUV frames is pretty frequent when it comes to video streams. For instance, frames coming from a camera or frames resulting from the decoding of compressed streams may be in some YUV format, for instance yuv420p. It happened to me quite some times that I needed to convert those frames to some RGB format and also render them in an OpenGL scene in an efficient way.

GPU

The simplest technique is to convert (as efficiently as possible) the frames to RGB and then upload the RGB frames to OpenGL. Another option is to upload YUV frames directly to the GPU and implement the conversion in an OpenGL shader, which should be even more efficient, and should also result in a lower bandwidth during the transfer. I created a simple demo app in Qt/OpenGL that does exactly this: https://github.com/carlonluca/qtyuv.

Shader

The computation is entirely done in the shader:

uniform sampler2DRect s_texture_y;
uniform sampler2DRect s_texture_u;
uniform sampler2DRect s_texture_v;
uniform lowp float qt_Opacity;
varying highp vec2 vTexCoord;

void main()
{
    float Y = texture2DRect(s_texture_y, vTexCoord).r;
    float U = texture2DRect(s_texture_u, vec2(vTexCoord.x/2., vTexCoord.y/2.)).r - 0.5;
    float V = texture2DRect(s_texture_v, vec2(vTexCoord.x/2., vTexCoord.y/2.)).r - 0.5;
    vec3 color = vec3(Y, U, V);
    mat3 colorMatrix = mat3(
                1,   0,       1.402,
                1,  -0.344,  -0.714,
                1,   1.772,   0);
    gl_FragColor = vec4(color*colorMatrix, 1.)*qt_Opacity;
}

The three planes in the yuv420p frame are uploaded to the GPU and the shader multiplies the vector by a conversion matrix:

$$\left[\begin{array}{c}R\\ G\\ B\end{array}\right]=\left[\begin{array}{ccc}1& 0& 1.402\\ 1& -0.344& -0.714\\ 1& 1.772& 0\end{array}\right]\cdot \left[\begin{array}{c}Y\\ U\\ V\end{array}\right]$$

Demo

In https://github.com/carlonluca/qtyuv there is a demo written in C++ using Qt5. If you want to run it, first create some yuv420p stream file. If you want, you can create it quickly with ffmpeg by transcoding the small sample in the repo:

ffmpeg -i coffee_run.mkv -pix_fmt yuv420p -f rawvideo coffee_run.raw

The result is around 2GB, so I did not include it in the repo.

Then, you’ll have to build the demo (I tested with Qt 5.15.2):

mkdir build
cd build
cmake ..
make

The you can run it:

./Qt5Yuv ../coffee_run.raw 2048x858 24

If you pass -h you’ll see what the options mean:

./Qt5Yuv --help
Usage: ./Qt5Yuv [options] source size fps
Test helper

Options:
  -h, --help     Displays help on commandline options.
  --help-all     Displays help including Qt specific options.
  -v, --version  Displays version information.

Arguments:
  source         Raw yuv420p file
  size           Size with the format widthxheight
  fps            Framerate in frames per second

Have fun!