Synchronized audio mixing in GStreamer

Over the last few weeks I worked on a new GStreamer element: audiomixer. This new element is based on adder, i.e. it mixes multiple audio streams together and produces a single audio stream. It’s already merged into GIT master of the gst-plugins-bad module.

The main and important difference to adder is that it actually synchronizes the different audio streams against each other instead of just mixing samples together as they come while ignoring all timing information. This was a feature that was requested since a long time and in practice causes problems if streams are supposed to start at different times, have gaps or slightly different clock rates (and thus one is supposed to run a bit faster than the other). It’s also an important first step to properly support mixing of live streams. The video element for mixing, videomixer, properly implements synchronization since a few years now.

A very simple example to see the difference between both elements would be the following gst-launch command:

gst-launch-1.0 audiomixer name=mix
  mix. ! audioconvert ! audioresample ! autoaudiosink
  audiotestsrc num-buffers=400 volume=0.2 ! mix.
  audiotestsrc num-buffers=300 volume=0.2 freq=880 timestamp-offset=1000000000 ! mix.
  audiotestsrc num-buffers=100 volume=0.2 freq=660 timestamp-offset=2000000000 ! mix.

If you replace audiomixer by adder, you’ll hear all streams starting at the same time while with audiomixer they start with the correct offsets to each other.

So, what’s left to be done. Currently reverse playback/mixing is not support, that’s somewhere next on my todo list. Also the handling of flushing seeks and flushes in general on mixers (and muxers) is currently rather suboptimal, that’s something I’m working on next. As a side-effect this will also bring us one step nearer to proper mixing of live streams.

16 thoughts on “Synchronized audio mixing in GStreamer”

  1. Very nice work. Don’t forget to show this to WebKit developers, they might be interested, and here is why I think so. Currently, if a bad guy places 32 invisible audio elements that just play silence on a web page, WebKit will open 32 streams with PulseAudio. This amounts to a complete DoS of all sound in the system, because PulseAudio will not allow any more connections from other applications. Pre-mixing sounds from one tab will prevent a single tab from doing such a DoS.

    1. Yes, it’s 1.0+ only. Shouldn’t be too difficult to backport to 0.10 though, but why would you even want that? 😉
      What’s this music mixer application you’re talking about? Is the code available somewhere already to try it? 🙂

      1. Because I’m too lazy/timedeprived to port to GI and GTK+3 all at once, and if I do that for one thing I’m going to feel I need to do it for other apps too…

        There is code, yes, but I don’t remember what state I left it in before my life took over:
        https://github.com/thomasvs/dad/

        There’s a mixing backend using gnonlin, and another that just uses websockets and in-browser playing, but that triggered all sorts of bugs in browser implementations.

      1. You install all the build dependencies, e.g. via apt-get, then clone the git repositories and run ./autogen.sh, make and then make install. There should be many instructions on the Internet about compiling software in general, and even some about compiling GStreamer in specific. The documentation from the Pitivi project here might help for example: http://wiki.pitivi.org/wiki/Building_with_GES

  2. I’ve finally installed gstreamer 1.3.x.x and gst-plugins-base. Now I’m trying to install gst-plugins-bad-master where is audiomixer. When I execute the autogen script in cmd in the end i see
    configure: *** Plug-ins without external dependencies that will be built:
    … audiomixer …
    but in sudo make operation I see
    ———————————————————————-
    make[3]: Leaving directory /home/virtual1/Desktop/gst-plugins-bad-master/gst/audiofxbad'
    make[2]: Leaving directory
    /home/virtual1/Desktop/gst-plugins-bad-master/gst/audiofxbad’
    Making install in audiomixer
    make[2]: Entering directory /home/virtual1/Desktop/gst-plugins-bad-master/gst/audiomixer'
    make install-am
    make[3]: Entering directory
    /home/virtual1/Desktop/gst-plugins-bad-master/gst/audiomixer’
    make[4]: Entering directory /home/virtual1/Desktop/gst-plugins-bad-master/gst/audiomixer'
    make[4]: Nothing to be done for
    install-exec-am’.
    /bin/mkdir -p ‘/usr/local/lib/gstreamer-1.0’
    /bin/bash ../../libtool –mode=install /usr/bin/install -c libgstaudiomixer.la ‘/usr/local/lib/gstreamer-1.0’
    libtool: install: /usr/bin/install -c .libs/libgstaudiomixer.so /usr/local/lib/gstreamer-1.0/libgstaudiomixer.so
    libtool: install: /usr/bin/install -c .libs/libgstaudiomixer.lai /usr/local/lib/gstreamer-1.0/libgstaudiomixer.la
    libtool: finish: PATH=”/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/sbin” ldconfig -n /usr/local/lib/gstreamer-1.0

    what could be wrong?

    1. What exactly is the problem? The plugin is built and properly installed according to that output, into /usr/local.

      If GStreamer does not find it, are you using your distro’s installation (in /usr) of GStreamer or do you have your own built version installed into /usr/local too? The latter should work without problems, for the former you’ll either need to move libgstaudiomixer.so to /usr/lib/gstreamer-1.0 (or /usr/lib/*/gstreamer-1.0), or use the GST_PLUGIN_PATH environment variable.

      1. More clear description of problem. I have two files in /usr/local/lib/gstreamer-1.0 needed to execute audiomixer element called: libgstaudiomixer.la and libgstaudiomixer.so. I linked this fol;der in bashsr file using export $GST_PLUGIN_PATH=/usr/local/lib/gstreamer-1.0, also I copied this two files to /usr/lib/i386-linux-gnu/gstreamer-1.0 (also add this path to bashsr file) and when I’m trying to run example command:

        gst-launch-1.0 audiomixer name=mix
        mix. ! audioconvert ! audioresample ! autoaudiosink
        audiotestsrc num-buffers=400 volume=0.2 ! mix.
        audiotestsrc num-buffers=300 volume=0.2 freq=880 timestamp-offset=1000000000 ! mix.
        audiotestsrc num-buffers=100 volume=0.2 freq=660 timestamp-offset=2000000000 ! mix.

        it still didn’t add:
        (gst-launch-1.0:3579): GStreamer-WARNING **: Failed to load plugin ‘/usr/local/lib/gstreamer-1.0/’: /usr/local/lib/gstreamer-1.0/.so: cannot open shared object file: No such file or directory
        ERROR: pipeline could not be constructed: no element “audiomixer”.
        ./audiomixer_ex.sh: 2: ./audiomixer_ex.sh: mix.: not found
        ./audiomixer_ex.sh: 3: ./audiomixer_ex.sh: audiotestsrc: not found
        ./audiomixer_ex.sh: 4: ./audiomixer_ex.sh: audiotestsrc: not found
        ./audiomixer_ex.sh: 5: ./audiomixer_ex.sh: audiotestsrc: not found

      2. The first problem seems to be that something in your file system is wrong around /usr/local/lib/gstreamer-1.0. Maybe a link to a non-existing directory, or you have a file named “.so” in there or something like that.

        And then you’ll have to add \ at the end of the lines in your shell script to have the multiple lines executed as a single command. If you do that and copy the libgstaudiomixer.so into /usr/lib/…/gstreamer-1.0 it should work. Otherwise please set GST_DEBUG=6 in your environment and send me a debug log.

    1. It looks like your libgstaudiomixer.so is blacklisted for whatever reason, the one from /usr/lib/i386-linux-gnu/gstreamer-1.0/libgstaudiomixer.so
      Can you try removing ~/.cache/gstreamer-1.0/registry.* and then for the very next run send me the log?

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.