Month: August 2014

Concatenate multiple streams gaplessly with GStreamer

Earlier this month I wrote a new GStreamer element that is now integrated into core and will be part of the 1.6 release. It solves yet another commonly asked question on the mailing lists and IRC: How to concatenate multiple streams without gaps between them as if they were a single stream. This is solved by the concat element now.

Here are some examples about how it can be used:

# 100 frames of the SMPTE test pattern, then the ball pattern
gst-launch-1.0 concat name=c ! videoconvert ! videoscale ! autovideosink  videotestsrc num-buffers=100 ! c.   videotestsrc num-buffers=100 pattern=ball ! c.

# Basically: $ cat file1 file2 > both
gst-launch-1.0 concat name=c ! filesink location=both   filesrc location=file1 ! c.   filesrc location=file2 ! c.

# Demuxing two MP4 files with h264 and passing them through the same decoder instance
# Note: this works better if both streams have the same h264 configuration
gst-launch-1.0 concat name=c ! queue ! avdec_h264 ! queue ! videoconvert ! videoscale ! autovideosink   filesrc location=1.mp4 ! qtdemux ! h264parse ! c.   filesrc location=2.mp4 ! qtdemux ! h264parse ! c.

If you run this in an application that also reports time and duration you will see that concat preserves the stream time, i.e. the position reporting goes back to 0 when switching to the next stream and the duration is always the one of the current stream. However the running time will be continuously increasing from stream to stream.

Also as you can notice, this only works for a single stream (i.e. one video stream or one audio stream, not a container stream with audio and video). To gaplessly concatenate multiple streams that contain multiple streams (e.g. one audio and one video track) one after another a more complex pipeline involving multiple concat elements and the streamsynchronizer element will be necessary to keep everything synchronized.

Details

The concat element has request sinkpads, and it concatenates streams in the order in which those sinkpads were requested. All streams except for the currently playing one are blocked until the currently playing one sends an EOS event, and then the next stream will be unblocked. You can request and release sinkpads at any time, and releasing the currently playing sinkpad will cause concat to switch to the next one immediately.

Currently concat only works with segments in GST_FORMAT_TIME and GST_FORMAT_BYTES format, and requires all streams to have the same segment format.

From an application side you could implement the same behaviour as concat implements by using pad probes (waiting for EOS) and using pad offsets (gst_pad_set_offset()) to adjust the running times. But by using the concat element this should be a lot easier to implement.

GStreamer Playback API

Update: the code is now also available on GitHub which probably makes it easier for a few people to use it and contribute. Just send pull requests or create issues in the issue tracker of GitHub.

Over the last years I noticed that I was copying too much code to create simple GStreamer based playback applications. After talking to other people at GUADEC this year it was clear that this wasn’t only a problem on my side but a general problem. So here it is, a convenience API for creating GStreamer based playback applications: GstPlayer.

The API is really simple but is still missing many features:

GstPlayer *  gst_player_new       (void);

void         gst_player_play      (GstPlayer * player);
void         gst_player_pause     (GstPlayer * player);
void         gst_player_stop      (GstPlayer * player);
void         gst_player_seek      (GstPlayer * player, GstClockTime position);
void         gst_player_set_uri   (GstPlayer * player, const gchar * uri);
...

Additionally to that there are a few other properties (which are not only exposed as setters/getters but also as GObject properties), and signals to be notified about position changes, errors, end-of-stream and other useful information. You can find the complete API documentation here. In general the API is modeled similar to other existing APIs like Android’s MediaPlayer and iOS’ AVPlayer.

Included are also some very, very simple commandline, GTK+, Android (including nice Java bindings) and iOS apps. An APK for the Android app can also be found here. It provides no way to start a player, but whenever there is a video or audio file to be played it will be proposed as a possible application via the Android intent system.

In the end the goal is to have a replacement for most of the GStreamer code in e.g. GNOME‘s Totem video player, Enlightenment‘s Emotion or really any other playback application, and then have it integrated in a gst-plugins-base library (or a separate module with other convenience APIs).

While this is all clearly only the start, I hope that people already take a look at this and consider using it for their projects, provide patches, or help making the included sample apps really useful and nice-looking. Apps for other platforms (e.g. a Qt app, or one written in other languages like C# or Python) would also be nice to have. And if you’re an Android or iOS or Qt developer and have no idea about GStreamer you can still help by creating an awesome user interface 🙂 Ideally I would like to get the Android and iOS app into such a good shape that we can upload them to the app stores as useful GStreamer playback applications, which we could then also use to point people to a good demo.

If you’re interested and have some time to work on it or try it, please get in contact with me.