OpenGL support in GStreamer

Over the last few months Matthew Waters, Julien Isorce and to some lesser degree myself worked on integrating proper OpenGL support into GStreamer.

Previously there were a few sinks based on OpenGL (osxvideosink for Mac OS X and eglglessink for Android and iOS), but they all only allowed rendering to a window. They did not allow rendering of a video into a custom texture that is then composited inside the application into an OpenGL scene. And then there was gst-plugins-gl, which allowed more flexible handling of OpenGL inside GStreamer pipelines, including uploading and downloading of video frames to the GPU, provided various filters and base classes to easily implement shader-based filters, provided infrastructure for sharing OpenGL contexts between different elements (even if they run in different threads) and also provided a video sink. The latter was now improved a lot, ported to all the new features for hardware integration and finally merged into gst-plugins-bad. Starting with GStreamer 1.4 in a few weeks, OpenGL will be a first-class citizen in GStreamer pipelines.

After yesterday’s addition of EAGL support for iOS (EAGL is Apple’s iOS API for handling GLES contexts), there is nothing missing to use this new set of library and plugins on all platforms supported by GStreamer. And finally we can get rid of eglglessink, which was only meant as an intermediate solution until we have all the infrastructure for real OpenGL support.

16 thoughts on “OpenGL support in GStreamer”

  1. Dear Slomo,

    i am trying to change a glshader element at runtime in a native Gstreamer application.

    The idea is to convert a rtsp Stream to an side-by-side view for a android google cardboard VR device.
    I want to implement headtracking as well and therefore want to change the opengl fragment dynamically based on head position.

    How can i change for example DYNAMICALLY during runtime in the native code?

    There used to be something like gstglfiltershader, but i can not find a similar functionality in the current 1.4.x releases.

    Thanks!!!!
    Kind regards
    Christian

  2. Is glupload available with the latest version of Cinder on iOS? I’m trying to attempt to put the output of a GStreamer pipeline into an OpenGL texture inside a Cinder application (libcinder.org).

    This call returns a null pointer :
    GstElement * mGlupload = gst_element_factory_make(“glupload”, NULL);

    I can create a glimagesink however. Is there a preferred approach that you know of in the latest version of GStreamer to accomplish this?

    1. No, it’s not a separate element anymore but automagically happens when you use a GL element. There’s a GstGLUpload and GstGLDownload object inside libgstgl though, which is internally and transparently used by glimagesink and others.

      For integration with other GL based libraries, take a look at http://cgit.freedesktop.org/gstreamer/gst-plugins-bad/tree/tests/examples/gl or https://github.com/ystreet/gtkgst . There are many somewhat convenient ways now how you can integrate GStreamer with GL based libraries.

  3. Is it possible to use the opengl plugins without X (in a headless server) ?
    For example a pipeline like this (no X running)
    filesrc ! matroskademux ! vp8dec ! gleffects_mirror ! vp8enc ! rtpvp8pay ! udpsink

  4. Hello!
    Can you help me with GStreamer?
    I have GStreamer 1.5.0 (GIT). I try to build Tutorial 5 from here:http://cgit.freedesktop.org/~slomo/gst-sdk-tutorials/tree/gst-sdk/tutorials/xcode%20iOS/Tutorial%205

    Build failed due to next errors:
    0:00:04.700213000 [334m52605[00m 0x3941040 [31;01mERROR [00m [00;01;37;41m GST_PIPELINE grammar.y:691:int priv_gst_parse_yyparse(void *, graph_t *):[00m no element “eglglessink”
    0:00:04.702176000 [334m52605[00m 0x3941040 [31;01mERROR [00m [00;01;37;41m GST_PIPELINE grammar.y:776:int priv_gst_parse_yyparse(void *, graph_t *):[00m link has no sink [source=@0x3959098]
    2015-08-28 15:53:33.582 MyCompany[52605:3321985] Pipeline error Unable to build pipeline: no element “eglglessink”

    1. There’s nothing in the tutorial code anywhere that creates an eglglessink. Do you have any local changes or a very old version or so? The new video sink on iOS is glimagesink.

      1. Thank you! It was old code from not fresh github repo.
        I have replaced old GStreamerBackend now with new.
        Now it starts without errors and change my state label to “State changed to PLAYING”, but my uiview is empty. I am trying to play rtsp wowza example: rtsp://wowzaec2demo.streamlock.net/vod/mp4:BigBuckBunny_115k.mov

        Here is GStreamerBackend log:
        http://pastebin.com/NYzMQvJF

        Thank you!
        ps: I don’t want to use ffmpeg for streaming from ip cameras. 😉

Leave a Reply

Your email address will not be published. Required fields are marked *