In the last days I was working at Centricular on adding PTP clock support to GStreamer. This is now mostly done, and the results of this work are public but not yet merged into the GStreamer code base. This will need some further testing and code review, see the related bug report here.
You can find the current version of the code here in my freedesktop.org GIT repository. See at the very bottom for some further hints at how you can run it.
So what does that mean, how does it relate to GStreamer?
Precision Time Protocol
PTP is the Precision Time Protocol, which is a network protocol standardized by the IEEE (IEEE1588:2008) to synchronize the clocks between different devices in a network. It’s similar to the better-known Network Time Protocol (NTP, IETF RFC 5905), which is probably used by millions of computers down there to automatically set the local clock. Different to NTP, PTP promises to give much more accurate results, up to microsecond (or even nanosecond with PTP-aware network hardware) precision inside appropriate networks. PTP is part of a few broadcasting and professional media standards, like AES67, RAVENNA, AVB, SMPTE ST 2059-2 and others for inter-device synchronization.
PTP comes in 3 different versions, the old PTPv1 (IEEE1588-2002), PTPv2 (IEEE1588-2008) and IEEE 802.1AS-2011. I’ve implemented PTPv2 via UDPv4 for now, but this work can be extended to other variants later.
GStreamer network synchronization support
So what does that mean for GStreamer? We are now able to synchronize to a PTP clock in the network, which allows multiple devices to accurately synchronize media to the same clock. This is useful in all scenarios where you want to play the same media on different devices, and want them all to be completely synchronized. You can probably imagine quite a few use cases for this yourself now, especially in the context of the “Internet of Things” but also for more normal things like video walls or just having multiple screens display the same thing in the same room.
This was already possible previously with the GStreamer network clock, but that clock implements a custom protocol that only other GStreamer applications can understand currently. See for example here, here or here. With the PTP clock we now get another network clock that speaks a standardized protocol and can interoperate with other software and hardware.
Performance, WiFi and other unreliable networks
When running the code, you will probably notice that PTP works very well in controlled and reliable networks (2-20 microseconds accuracy is what I got here). But it’s not that accurate in wireless networks or in general unreliable networks. It seems like in those networks the custom GStreamer network clock protocol works more reliable currently, partially by design.
Future
As a next step, at Centricular we’re going to look at implementing support for RFC7273 in GStreamer, which allows to signal media clocks for RTP. This is part of e.g. AES67 and RAVENNA and would allow multiple RTP receivers to be perfectly synchronized against a PTP clock without any further configuration. And just for completeness, we’re probably going to release a NTP based GStreamer clock in the near future too.
Running the code
If you want to test my code, you can run it for example against PTPd. And if you want to test the accuracy of the clock, you can measure it with the ptp-clock-reflector (or here, instructions in the README) that I wrote for testing. The latter allows you to measure the accuracy, and in a local wired network I got around 2-20 microseconds accuracy. A GStreamer example application can be found here, which just prints the local and remote PTP clock times. Other than that you can use it just like any other clock on any GStreamer pipeline you can imagine.
Too long ago to remember where I read it, someone else was working with network audio synchronization. They also encountered problems with PTP over wifi and powerline networks. If I recall correctly, they found that NTP was unexpectedly the superior solution for network audio synchronization. It may have been Lennart Poettering working on Pulse Audio that observed this. I’m interested to see your results after implementing NTP.
The custom protocol of the GStreamer network clock implements something very similar to NTP, so I can confirm that NTP (-like) protocols seem to work better over wifi.
Thanks, any work in this area is really appreciated!
Can you give a little bit more detail about your setup? Network topology, used NICs? Do your NICs support HW timestamping (ethtool -T )? Do your switches support transparent clocks (i.e. reporting of residence time)?
The biggest problem with 802.11/WLAN is caused by its shared medium nature. Any PTP packet may be delayed by a hugely varying amount of time, and if timestamping on the ingress side happens in software, you will be also affected by varying data rates.
As NTP and PTP use the same mechanisms, PTP should be at least as good as NTP. If PTP is inferior, the culprit most likely is the filtering.
I have observed PTP destabilizing over networks with random large delays – powerline networking especially, while NTP continues to function. Maybe NTP implementations are throwing out obviously wrong packets and PTP isn’t? It is almost like you need to make a histogram of the incoming packets, locate the fastest grouping, and then ignore the rest.
See my other comment here. I already implemented some filtering (inspired by the filtering done in the custom GStreamer network clock) and still have some further ideas to experiment with. In the worst case, PTP could fall back to a NTP-style behaviour.
The problem with PTP is that clock synchronization and path delay measurement are not as closely correlated in time as with NTP. You get SYNC/FOLLOW_UP messages for clock synchronization but have no idea how much delayed it is, and then send DELAY_REQ (after a random delay that can be relatively big) and receive DELAY_REQ to measure the path delay. However you are not necessarily allowed to measure the path delay for every single message due to the timing rules.
So you might synchronize to the remote clock from the SYNC/FOLLOW_UP message without knowing how much it was delayed. And even if you know how much the SYNC/DELAY_RESP together was delayed, there could be a big difference between the SYNC delay and the DELAY_REQ/RESP delay.
If you compare that to NTP, there you send a request, get a reply ASAP and can measure the delay for the these two packets and use the times of these two packets. The delay you measure is measured much closer to the times in the packets.
Now you might say that you could just do the same as with NTP for the DELAY_REQ/DELAY_RESP and that would be true, but that’s not what the standard specifies. Or you could also filter many outliers based on the different timings you can observe and estimations of the remote clock, and by not just taking the latest delay measurement but keeping a running average (which is what I implemented btw and which improves the accuracy over wifi quite a bit). And instead of taking just one time measurement, you could take multiple and run a linear regression over them (which is also what I do). But that’s also not what the standard says you should do 🙂 I also have some further ideas how the filtering could be improved.
I think the main problem with PTP for networks like wifi, where every packet can be delayed completely differently, is that you have no idea how much the SYNC messages are delayed. You basically can’t trust them at all in unreliable networks, and would have to fall back to NTP-style behaviour based on DELAY_REQ/RESP only. That, and some possible improvements to the filtering, are what I’m going to experiment with next.
If you have any pointers, where else I should look at for further ideas, please let me know 🙂
What is pulse audio doing? I know it synchronizes the clocks somehow and then uses the RTP timestamps and RTP multicast.
Anything useful here?
https://www.eecis.udel.edu/~mills/ntp.html
Too bad we can’t get to the zero crossing information from the AC powerline.
Thanks, I’ll check if there’s something useful there in the next days 🙂 There’s in any case a lot of opportunity for various tuning and heuristics, you can easily spend years on “time” 😉
RTP timestamps alone don’t help you much as they have random (really random by design) offsets, and you would have to get a mapping between the RTP time and some shared clock between different devices, i.e. you have to find the RTP timestamp offset and the relative rate between your audio hardware clock and the sender clock.
FWIW, the accuracy I got over wifi here should be enough for synchronized audio playback on different devices. I’ll test it with actual audio on Monday, instead of just collecting statistics and will let you know.
There may be some experts hanging out on the PTPd mailing list
http://sourceforge.net/p/ptpd/mailman/
ELEC did a new video on PTP
https://www.youtube.com/watch?v=pNiOlRi8jK4&list=PLGeM09tlguZTPUxEvsQiDgX0XDjfOL6oR&index=37
This looks really promising. Is there any container format (.mov, .mp4, .mkv, etc…) that can save video, ideally uncompressed or lossless compressed, with high resolution timestamps such as from PTP?
Matroska and mp4 use a fraction for the timestamp, so that would be possible. But the accuracy there should not be needed for most use cases.
Which use case are you thinking of?
In my lab we record a lot of video of animal behavior using multiple synchronized cameras. Historically I’ve stayed away from normal containers and codecs because my difficulty in knowing how to associate precise timestamps referenced to an external clock with each frame. (Some of our cameras can occasionally fail to deliver a frame, although they are triggered at regular intervals.) It would be really great if we could use standard video containers to store our videos, but so far I haven’t thought this was possible. I would really appreciate any pointers into how to do this.
I’m not sure I understand where the problem is, this should just work. Just might cause some problems if timestamps don’t start near zero but instead at 45 years and a bit 🙂 You’ll just have to try and see into which problems you run, and then fix those. There’s no inherent reason why this shouldn’t work.
What’s the plan for 1.6’s release? I’m having trouble finding information about it. It seems like it has been in git master for a while.
There was a 1.5.1 release on this Sunday, which included the PTP clock (and also an NTP clock). 1.6 is planned to be released soonish now, ideally some time in July. Depends a bit on how much testing and fixing of blocker bugs happens.
Awesome. Thanks for the info! I still need to submit bug reports for the two seeking-related bugs I discovered a few months ago when using your concat element. Seeking a pipeline (or bin) using a concat element while paused deadlocks the main thread and seeking a pipeline using a concat element is inaccurate and always seeks farther than it should. I plan to do it soon. I just need to figure out how to compile GStreamer with full debug symbols on Arch and make some test cases.
Please report 🙂 This might be related to https://bugzilla.gnome.org/show_bug.cgi?id=745366 or https://bugzilla.gnome.org/show_bug.cgi?id=750027
Ah I forgot to add the link to the release announcement: http://lists.freedesktop.org/archives/gstreamer-devel/2015-June/053070.html
Both bugs seem to be fixed with today’s release. Thanks for the information and your support of GStreamer!
Can’t tell – does your PTP implementation require ptpd, or does it reimplement the protocol?
Would ptpd running disrupt the Gstreamer version in that case? Can Gstreamer be a master so that the output clock (vertical refresh) can be used as a master?
Thanks!
It’s client-side only, you need to run ptpd or another implementation (or some hardware box) somewhere in your network and then GStreamer can sync against that.
Hi,
Is there an example application for Gstreamer, that demonstrates synchronization using a PTP clock from a hardware and / or PTPd running on a machine and that gives example of audio multicast and sync (one master that send audio packed via PTP and say two slave computers that sync agains the master and play the audio in sync)?
P.S (I only discovered the example that prints master / slave times)
Thank You!
You could take for example these examples [client] [server] and instead of a netclock you create a PTP clock. It works the same with all GStreamer clocks.
Thank you for the fast reply.
Hello.
I tested ptp clock implementation with your print diff program & reflector with two native ubuntu desktops. It seems that reflector makes domain 1 available and ptp clock element syncs with domain 1, not domain 0 (for ptpd)
I have a question.
Is it possible to test it using ptpd locally? (loopback?)
For example, when I run the ptpd with domain 0, is it possible to sync the test program(print diff) to domain 0 in same workstation?
Thank you.
No because both the ptpd and GStreamer will have to bind to port 123, and the ptpds are usually not doing that with SO_REUSEPORT
123 port is right?
As I know, 319, 320 are used in PTP.
Thank you for your reply.
Yes, sorry. 123 is NTP 🙂 Same problem though
Just curious. If I’m not wrong, there is no NTP support as such in Gstreamer. That is because the system is supposed to sync with NTP clock and then the gstreamer uses System clock as the master clock
Same way, if there is something like ptpd which makes system synchronize through PTP, is there a need to do this within gstreamer?
There is NTP client clock support in GStreamer: https://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer-libs/html/GstNetClientClock.html#gst-ntp-clock-new
Using the GStreamer clocks for this allows you to use NTP/PTP for media synchronization without also having to synchronize your system clock, and you could e.g. use different clocks for different streams.
Hello,
as I see more and more network cards has support for PTP Hardware stamping.
It is possible to use this Hardware stamping with the Gstreamer clock?
Currently it’s not making use of that, but that could be added relatively easily by someone who needs it.