Synchronized audio mixing in GStreamer

Over the last few weeks I worked on a new GStreamer element: audiomixer. This new element is based on adder, i.e. it mixes multiple audio streams together and produces a single audio stream. It’s already merged into GIT master of the gst-plugins-bad module.

The main and important difference to adder is that it actually synchronizes the different audio streams against each other instead of just mixing samples together as they come while ignoring all timing information. This was a feature that was requested since a long time and in practice causes problems if streams are supposed to start at different times, have gaps or slightly different clock rates (and thus one is supposed to run a bit faster than the other). It’s also an important first step to properly support mixing of live streams. The video element for mixing, videomixer, properly implements synchronization since a few years now.

A very simple example to see the difference between both elements would be the following gst-launch command:

gst-launch-1.0 audiomixer name=mix
  mix. ! audioconvert ! audioresample ! autoaudiosink
  audiotestsrc num-buffers=400 volume=0.2 ! mix.
  audiotestsrc num-buffers=300 volume=0.2 freq=880 timestamp-offset=1000000000 ! mix.
  audiotestsrc num-buffers=100 volume=0.2 freq=660 timestamp-offset=2000000000 ! mix.

If you replace audiomixer by adder, you’ll hear all streams starting at the same time while with audiomixer they start with the correct offsets to each other.

So, what’s left to be done. Currently reverse playback/mixing is not support, that’s somewhere next on my todo list. Also the handling of flushing seeks and flushes in general on mixers (and muxers) is currently rather suboptimal, that’s something I’m working on next. As a side-effect this will also bring us one step nearer to proper mixing of live streams.