Timecodes

Timecode overview and requirements

The purpose of this document is to communicate what timecodes are and why they are needed as part of an Ease Live solution, but also to provide insight into the challenges in end-to-end transport of timecodes and methods on how to solve them.

What differentiates Ease Live from traditional broadcast solutions is the rendering of overlay graphics on the consumer device instead of being rendered into the video picture. This creates a suite of new opportunities, and requires correctly timing the overlay graphic to the right point in the video stream. We call these timing points ‘events’ and when referring to events we mean any data sent to the consumer that needs to be synced up to the video. Examples of this can be statistics, news items, social media, or any change in overlay graphics triggered by an operator either in manual or in an automated fashion. Events can also cover data that does not need sync with the video stream, in this document we will focus on the event domain space that do.

To enjoy the full range of flexibility of aggregating multiple data sources at the consumer end, scaling and curating of data streams, we have different solutions in place to facilitate event and video synchronisation.

It’s essential to provide timing information to each data source, both the video stream and any stream of event that is to be synced with it. With this in place any shift latency delivery, either being for the many processes that makes up a delivery pipeline for any given data stream, or from internet induced latency because of it's unpredictable nature will be remedied when the streams are synced up on the consumer device.

What are timecodes?

Timecodes are any representation of clock information in a data stream, type or format is non essential as long as they can be read and understood where they are aggregated and synced, this usually on the consumer device. What is important however is that all data streams that are to be aggregated are on the same timeline. Or to put it another way, it's not sufficient to use the same clock source if there are timeshift in any of the data streams from misalignment in where the clock data is either read or written. This is best explained using an illustration.

Steps in timecode transport

The preferred solution is to have all events sources and video streams carry timecodes, and for this to happen three things have to be in place:

1. Events and video must use the same clock as a source for its timing data
2. The timing data must be transported unmodified from ingestion point to the client
3. Event creation must align with the timing data in the video stream

Ideal scenario

The effect of this however is that the insertion of timecodes into the video stream often has to be moved upstream, closer to the point where the events themselves are generated. In this also comes the challenge to carry timecodes along the video through all the processes that make up the video delivery pipeline from the broadcaster to internet relied consumption, the last part usually being an adaptive streaming format: dash / hls / smooth.

In the broadcasting side that usually means effort has to be made to ensure all equipment taking part of the video pipeline has to support and carry timecode data so it's not lost or modified in undesirable ways along its path.

End-to-end transport of timecodes in the video stream is the ideal scenario, as it in theory would facilitate any event creation or curation along the value chain, this may not always be feasible for a numerous of reasons. There are, however, ways to circumvent many of these obstacles, though sometimes with a tradeoff in flexibility or accuracy.

Within the broadcast

In the broadcast production timecodes are a natural part of all video streams, and there are several systems available to synchronise timecodes between different equipment and video sources. Most broadcasters use a central timecode server source, often referred to as the ‘house timecode’ generator. These can be used for embedding timecode into video TC tracks, or to be accessed remotely for other services or remote OB productions.

This timecode is what we refer to as the ‘Origin of timecode’ as it’s normally within the broadcast production environment both the video streams and the data events are being generated. Sometimes data events can be generated elsewhere with other timecode stamps, and in such cases the offset of these will be aligned with the video timecode on the Ease Live Server. House timecode generators can also be used as a source to stamp data events outside the broadcast environment, including on the Ease Live Server.

The broadcast timecode should normally be carried in the master SDI/IP video in SMPTE format, to be included in the different transcode operations, to make sure the timecode is read by the encoders and included in the output video streams.

Ease Live currently supports the following formats and players, in order to synchronise with the timecode:

Streaming formats	DRM video players
HLS Supported	JW Player Supported
DASH Supported	BitMovin Supported
Smooth Supported	AVPlayer (iOS) Supported
	ExoPlayer (Android) Supported
Other formats can be supported on request.	Other video players can be integrated on request

OTT video stream synchronisation

Manifest files

HLS / Dash and Smooth all have support through tags in the manifest file for specifying timing information for the video segments. In HLS, the EXT-X-PROGRAM-DATE-TIME tag can be used for specifying the timecode, but as the video segment can carry multiple frames, the tag will only relate to the first frame of each segment.

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:4
#EXT-X-MEDIA-SEQUENCE:252680
#EXT-X-PROGRAM-DATE-TIME:2018-06-26T10:36:52.400Z
#EXTINF:4.000,
master_1200_20180626T103652_252680.ts
#EXTINF:4.000,
master_1200_20180626T103656_252681.ts
#EXTINF:4.000,
master_1200_20180626T103700_252682.ts
#EXTINF:4.000,
master_1200_20180626T103704_252683.ts
#EXTINF:4.000,
master_1200_20180626T103708_252684.ts

This EXT-X-PROGRAM-DATE-TIME tag can be read on the player side and used as a timecode source, and the Ease Live Bridge will through frame interpolation get frame accurate event and video synchronisation.

Besides being a solution that covers all the major adaptive streaming platforms Manifest file tags has the added benefit of being agnostic to the format used for the video transport.

Metadata transport

Apple’s HTTP Live Streaming (HLS), has support for custom metadata packages to be injected into the stream and transported along with the video data. This is referred to as ID3 tags, and by using this method of delivery for timecode data, we have a way to ensure proper synchronisation of event data on the consumer device. This is a solution we have extensively tested in our lab to ensure tooling to create and transport id3 tags are in place and also to verify pickup and accuracy is supported on the major video players, both native and browser based.

The ID3 metadata tags supports multiple options for adding the timecode information. It can be added as comments information tag within a COMM frame:

ID39COMM/eng{"tc":"10:46:23:22","ut":"1530009983906"}

Or the timecode can be added into a General encapsulated object, within the GEOB frame, like this example:

ID3~GEOBtapplication/xmlsubstitution.plist<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
 <key>ad-list</key>
 <array>
  <dict>
   <key>name</key>
   <string>GameClock</string>
   <key>data</key>
   <string>{             &quot;utc&quot; : &quot;1513210237367&quot;, &#x09;&#x09;&#x09;&quot;provider&quot; : &quot;atc-ltc&quot;,             &quot;dayTimeLocal&quot; : &quot;07:10 PM&quot;,             &quot;dayTimeHHMMSS&quot; : &quot;19:10:37&quot;             }</string>
  </dict>
 </array>
</dict>
</plist>

With multiple options for specifying and adding the timecode information within the ID3 tags we get better synchronisation of the data and the video. Each setup is unique, and the alignment of clock sources for video and data events is a key factor to get the synchronisation right. With the right setup, the use of ID3 tags gives frame-accurate synchronisation.

Additional synchronisation solutions

If the end-to-end timecode transport cannot follow the ideal scenario, there are other synchronisation solutions, one being:

Server-side correction

If the timecode in the video stream is lost or incorrect, any new timecode added to the stream on the online video platform, or the existing incorrect timecode can be manually or automatically aligned with the original event timestamps, using a timecode offset setting on the Ease Live Server. It is a requirement to have a timecode of some sort present in the stream playing on the consumer device.

Summary

At the end of the day there are many methods that timing can be carried in the stream, all the way through to the video player.

Timecode options for HLS:

• EXT-X-PROGRAM-DATE-TIME in the M3U8 variant playlist
• timed ID3 tags in video segments if MPEG2-TS is used
• timed EMSG in video segments if CMAF is used

Timecode options for DASH:

• availabilityStartTime in the MPD manifest
• timed metadata in InbandEventStream in the MPD manifest
• timed EMSG in the video segments

Which method should I use?

It depends on what's supported by your streaming server, and what can be read by your video player.

Most players will support EXT-X-PROGRAM-DATE-TIME and availabilityStartTime.