PTP network clock support in GStreamer

In the last days I was working at Centricular on adding PTP clock support to GStreamer. This is now mostly done, and the results of this work are public but not yet merged into the GStreamer code base. This will need some further testing and code review, see the related bug report here.

You can find the current version of the code here in my freedesktop.org GIT repository. See at the very bottom for some further hints at how you can run it.

So what does that mean, how does it relate to GStreamer?

Precision Time Protocol

PTP is the Precision Time Protocol, which is a network protocol standardized by the IEEE (IEEE1588:2008) to synchronize the clocks between different devices in a network. It’s similar to the better-known Network Time Protocol (NTP, IETF RFC 5905), which is probably used by millions of computers down there to automatically set the local clock. Different to NTP, PTP promises to give much more accurate results, up to microsecond (or even nanosecond with PTP-aware network hardware) precision inside appropriate networks. PTP is part of a few broadcasting and professional media standards, like AES67, RAVENNA, AVB, SMPTE ST 2059-2 and others for inter-device synchronization.

PTP comes in 3 different versions, the old PTPv1 (IEEE1588-2002), PTPv2 (IEEE1588-2008) and IEEE 802.1AS-2011. I’ve implemented PTPv2 via UDPv4 for now, but this work can be extended to other variants later.

GStreamer network synchronization support

So what does that mean for GStreamer? We are now able to synchronize to a PTP clock in the network, which allows multiple devices to accurately synchronize media to the same clock. This is useful in all scenarios where you want to play the same media on different devices, and want them all to be completely synchronized. You can probably imagine quite a few use cases for this yourself now, especially in the context of the “Internet of Things” but also for more normal things like video walls or just having multiple screens display the same thing in the same room.

This was already possible previously with the GStreamer network clock, but that clock implements a custom protocol that only other GStreamer applications can understand currently. See for example here, here or here. With the PTP clock we now get another network clock that speaks a standardized protocol and can interoperate with other software and hardware.

Performance, WiFi and other unreliable networks

When running the code, you will probably notice that PTP works very well in controlled and reliable networks (2-20 microseconds accuracy is what I got here). But it’s not that accurate in wireless networks or in general unreliable networks. It seems like in those networks the custom GStreamer network clock protocol works more reliable currently, partially by design.

Future

As a next step, at Centricular we’re going to look at implementing support for RFC7273 in GStreamer, which allows to signal media clocks for RTP. This is part of e.g. AES67 and RAVENNA and would allow multiple RTP receivers to be perfectly synchronized against a PTP clock without any further configuration. And just for completeness, we’re probably going to release a NTP based GStreamer clock in the near future too.

Running the code

If you want to test my code, you can run it for example against PTPd. And if you want to test the accuracy of the clock, you can measure it with the ptp-clock-reflector (or here, instructions in the README) that I wrote for testing. The latter allows you to measure the accuracy, and in a local wired network I got around 2-20 microseconds accuracy. A GStreamer example application can be found here, which just prints the local and remote PTP clock times. Other than that you can use it just like any other clock on any GStreamer pipeline you can imagine.

23 thoughts on “PTP network clock support in GStreamer”

  1. Too long ago to remember where I read it, someone else was working with network audio synchronization. They also encountered problems with PTP over wifi and powerline networks. If I recall correctly, they found that NTP was unexpectedly the superior solution for network audio synchronization. It may have been Lennart Poettering working on Pulse Audio that observed this. I’m interested to see your results after implementing NTP.

    1. The custom protocol of the GStreamer network clock implements something very similar to NTP, so I can confirm that NTP (-like) protocols seem to work better over wifi.

  2. Thanks, any work in this area is really appreciated!

    Can you give a little bit more detail about your setup? Network topology, used NICs? Do your NICs support HW timestamping (ethtool -T )? Do your switches support transparent clocks (i.e. reporting of residence time)?

    The biggest problem with 802.11/WLAN is caused by its shared medium nature. Any PTP packet may be delayed by a hugely varying amount of time, and if timestamping on the ingress side happens in software, you will be also affected by varying data rates.

    As NTP and PTP use the same mechanisms, PTP should be at least as good as NTP. If PTP is inferior, the culprit most likely is the filtering.

    1. I have observed PTP destabilizing over networks with random large delays – powerline networking especially, while NTP continues to function. Maybe NTP implementations are throwing out obviously wrong packets and PTP isn’t? It is almost like you need to make a histogram of the incoming packets, locate the fastest grouping, and then ignore the rest.

      1. See my other comment here. I already implemented some filtering (inspired by the filtering done in the custom GStreamer network clock) and still have some further ideas to experiment with. In the worst case, PTP could fall back to a NTP-style behaviour.

    2. The problem with PTP is that clock synchronization and path delay measurement are not as closely correlated in time as with NTP. You get SYNC/FOLLOW_UP messages for clock synchronization but have no idea how much delayed it is, and then send DELAY_REQ (after a random delay that can be relatively big) and receive DELAY_REQ to measure the path delay. However you are not necessarily allowed to measure the path delay for every single message due to the timing rules.
      So you might synchronize to the remote clock from the SYNC/FOLLOW_UP message without knowing how much it was delayed. And even if you know how much the SYNC/DELAY_RESP together was delayed, there could be a big difference between the SYNC delay and the DELAY_REQ/RESP delay.

      If you compare that to NTP, there you send a request, get a reply ASAP and can measure the delay for the these two packets and use the times of these two packets. The delay you measure is measured much closer to the times in the packets.

      Now you might say that you could just do the same as with NTP for the DELAY_REQ/DELAY_RESP and that would be true, but that’s not what the standard specifies. Or you could also filter many outliers based on the different timings you can observe and estimations of the remote clock, and by not just taking the latest delay measurement but keeping a running average (which is what I implemented btw and which improves the accuracy over wifi quite a bit). And instead of taking just one time measurement, you could take multiple and run a linear regression over them (which is also what I do). But that’s also not what the standard says you should do 🙂 I also have some further ideas how the filtering could be improved.

      I think the main problem with PTP for networks like wifi, where every packet can be delayed completely differently, is that you have no idea how much the SYNC messages are delayed. You basically can’t trust them at all in unreliable networks, and would have to fall back to NTP-style behaviour based on DELAY_REQ/RESP only. That, and some possible improvements to the filtering, are what I’m going to experiment with next.

      If you have any pointers, where else I should look at for further ideas, please let me know 🙂

    1. Thanks, I’ll check if there’s something useful there in the next days 🙂 There’s in any case a lot of opportunity for various tuning and heuristics, you can easily spend years on “time” 😉

      RTP timestamps alone don’t help you much as they have random (really random by design) offsets, and you would have to get a mapping between the RTP time and some shared clock between different devices, i.e. you have to find the RTP timestamp offset and the relative rate between your audio hardware clock and the sender clock.

      FWIW, the accuracy I got over wifi here should be enough for synchronized audio playback on different devices. I’ll test it with actual audio on Monday, instead of just collecting statistics and will let you know.

  3. This looks really promising. Is there any container format (.mov, .mp4, .mkv, etc…) that can save video, ideally uncompressed or lossless compressed, with high resolution timestamps such as from PTP?

    1. Matroska and mp4 use a fraction for the timestamp, so that would be possible. But the accuracy there should not be needed for most use cases.

      Which use case are you thinking of?

      1. In my lab we record a lot of video of animal behavior using multiple synchronized cameras. Historically I’ve stayed away from normal containers and codecs because my difficulty in knowing how to associate precise timestamps referenced to an external clock with each frame. (Some of our cameras can occasionally fail to deliver a frame, although they are triggered at regular intervals.) It would be really great if we could use standard video containers to store our videos, but so far I haven’t thought this was possible. I would really appreciate any pointers into how to do this.

      2. I’m not sure I understand where the problem is, this should just work. Just might cause some problems if timestamps don’t start near zero but instead at 45 years and a bit 🙂 You’ll just have to try and see into which problems you run, and then fix those. There’s no inherent reason why this shouldn’t work.

  4. What’s the plan for 1.6’s release? I’m having trouble finding information about it. It seems like it has been in git master for a while.

    1. There was a 1.5.1 release on this Sunday, which included the PTP clock (and also an NTP clock). 1.6 is planned to be released soonish now, ideally some time in July. Depends a bit on how much testing and fixing of blocker bugs happens.

      1. Awesome. Thanks for the info! I still need to submit bug reports for the two seeking-related bugs I discovered a few months ago when using your concat element. Seeking a pipeline (or bin) using a concat element while paused deadlocks the main thread and seeking a pipeline using a concat element is inaccurate and always seeks farther than it should. I plan to do it soon. I just need to figure out how to compile GStreamer with full debug symbols on Arch and make some test cases.

    2. Both bugs seem to be fixed with today’s release. Thanks for the information and your support of GStreamer!

  5. Can’t tell – does your PTP implementation require ptpd, or does it reimplement the protocol?
    Would ptpd running disrupt the Gstreamer version in that case? Can Gstreamer be a master so that the output clock (vertical refresh) can be used as a master?

    Thanks!

    1. It’s client-side only, you need to run ptpd or another implementation (or some hardware box) somewhere in your network and then GStreamer can sync against that.

Leave a Reply

Your email address will not be published. Required fields are marked *