Timecode Accuracy

[cheng-chi]

Dear GoPro Labs team,
Thank you so much for making such a developer-friendly and hackable product!
I'm a Robotics PhD student at Columba University. My current research project requires me to accurately sync multiple GoPro Hero 10s with a PC's clock that drives our OptiTrack system.
My current approach is:

• Use this "improved" precise time QR code that runs at 60FPS to set GoPro's clock
• Record a video of another rolling QR code with a custom QR command format "#230801230421.500" which contains the precise time information, but will not trigger a GoPro clock reset
• Calibrate the constant offset caused by QR command recognition (similar to TCAL) using a python script that:
  • Read the "tmcd" track and decode the 4 bytes as a big-endian integer, interpreted as frames since midnight
  • Use opencv to detect the QR code from the first video frame and parse the QR command that starts with a "#"
  • Calculate the constant offset between "#" timestamp and "tmcd" timecode.

If my understanding is correct, setting GoPro's clock once, and then recording multiple videos of the "#" custom rolling QR code without rebooting GoPro, should yield very similar (within 1/60 sec) timecode offsets between these videos. However, in my experiment, I found the time code offset could vary up to 0.5 seconds. A reddit user seems to agree with my experience.

Am I reading the correct timecode? Is my understanding incorrect? Is this problem specific to GoPro Hero 10? (I'm happy to purchase Hero 11 if its timecode is more accurate)

[dnewman-gpsw]

This sounds wrong. For a start there is no command that starts with a #. The time sync is always oMYYMMDDHHMMSS.sss Timecode from QR sync should be within 30-50ms, at least in my own testing. I will retest on HERO10 tomorrow, as all Labs enabled cameras should be pretty good at this.

[cheng-chi]

Hi David, thank you so much for your swift reply! To clarify:

• Yes the actual time sync was indeed done with "oTYYMMDDHHMMSS.sss" command in step 1.
• In step 2, the intention is to record time information on the video frames without triggering a GoPro clock reset, to verify that the latency between "tmcd" timecode and the first frame is constant. Therefore I modified the timecode javascript to use a non-existent command.

What is the last GoPro model you have tested for timecode accuracy? I also have a GoPro Hero 9 available for testing on my side.

[dnewman-gpsw]

Sorry this took a while, the data mostly met with my expectation, and the timecode generally remained within a frame. These camera are not TCAL (calibrated) so that both average around -32 frame offset. The two HERO10 cameras shot 4k @ 30fps and where synced using only https://gopro.github.io/labs/control/precisiontime/ The sync was done about every 4 captures, from the data you can see where as over 1h20m the cameras will not drift too much.

CAM A display timecode error frames
GX019997 12:06:06:02 12:06:04:27 -35
GX019998 12:07:29:04 12:07:28:02 -32
GX019999 12:09:16:29 12:09:15:27 -32
GX010001 12:29:29:14 12:29:28:01 -43 (The only significant anomaly.)
GX010002 12:29:59:20 12:29:58:16 -34
GX010003 12:45:04:18 12:45:03:16 -32
GX010004 12:45:54:11 12:45:53:08 -33
GX010005 12:52:11:17 12:52:10:15 -32
GX010006 12:52:47:03 12:52:46:00 -33
GX010007 13:00:08:20 13:00:07:17 -33
GX010008 13:01:20:09 13:01:19:06 -33
GX010009 13:11:16:17 13:11:15:14 -33
GX010010 13:11:56:00 13:11:54:28 -32
GX010011 13:13:41:22 13:13:40:20 -32
GX010012 13:14:35:13 13:14:34:10 -33
GX010013 13:15:20:02 13:15:19:00 -33
GX010014 13:16:03:15 13:16:03:12 -33
GX010015 13:16:33:19 13:16:32:16 -33
GX010016 13:17:13:29 13:17:12:27 -32
GX010017 13:17:57:20 13:17:56:17 -33
GX010018 13:22:44:25 13:22:43:22 -33

CAM B display timecode error frames
GX019531 12:06:05:24 12:06:04:23 -31
GX019532 12:07:29:03 12:07:28:01 -32
GX019533 12:09:16:25 12:09:15:23 -32
GX019534 12:29:28:28 12:29:27:27 -31
GX019535 12:29:59:19 12:29:58:17 -32
GX019536 12:45:04:17 12:45:03:15 -32
GX019537 12:45:54:10 12:45:53:08 -32
GX019538 12:52:11:29 12:52:10:28 -31
GX019539 12:52:43:14 12:52:42:11 -33
GX019540 13:00:08:15 13:00:07:13 -32
GX019541 13:01:20:10 13:01:19:08 -32
GX019542 13:11:16:20 13:11:15:18 -32
GX019543 13:11:55:28 13:11:54:27 -31
GX019544 13:13:41:23 13:13:40:22 -31
GX019545 13:14:35:10 13:14:34:08 -32
GX019546 13:15:25:04 13:15:24:03 -31
GX019547 13:16:03:14 13:16:03:14 -31
GX019548 13:16:33:17 13:16:32:14 -33
GX019549 13:17:13:28 13:17:12:26 -32
GX019550 13:17:57:29 13:17:56:26 -33
GX019551 13:22:44:24 13:22:43:22 -32

Now for the anomaly. The 10-11 frame additional error surprised me, as I've never seen that (or data anything like your tests.) The error happened on file GX010001 (in a new folder 101GOPRO), it seem that creating a new clip was delayed by the creation of the new folder. This suggests, full, old or slow media, can have a impaction on how quickly files can be created, and the causing some loss of precision in the timecode (just a speculation at the moment.) But in general, the above data meets +/-1 frame timecode I've seen for all Labs enabled cameras.

[cheng-chi]

Hi @dnewman-gpsw, thank you so much for the test!
Is it possible to share the code (or just the general methodology) of how you calculated these frame offsets? It could be that I'm calculating the wrong thing.
Also, I recorded these videos in extended battery mode. Could that influence the frame offset?

[dnewman-gpsw]

I just filmed this https://www.shadertoy.com/view/ts3BW7 from the same computer that generated the QR Code time. All timecode was read by Premiere Pro. No code was used. You just check that your computed frames since midnight matches the timecode read by Premiere, DaVinci or FCP-X. Or use the free ffprobe

ffprobe D:\Temp\timecode\CamB\GX010001.MP4

look for:
  Stream #0:2[0x3](eng): Data: none (tmcd / 0x64636D74), 0 kb/s (default)
    Metadata:
      creation_time   : 2023-08-02T12:30:13.000000Z
      handler_name    : GoPro TCD
      timecode        : 12:29:28:01

[cheng-chi]

@dnewman-gpsw Thank you so much! I will try to replicate your test tomorrow!

[cheng-chi]

Hi @dnewman-gpsw
I was able to reproduce a timestamp accuarcy of +/- 1 frames using your method! Thank you!
There must be something wrong with my timecode calculation.

[cheng-chi]

For future reference, here I will explain how I end up interpreting timecode information from GoPro mp4.
@dnewman-gpsw Thank you so much for all the help! Please correct me if I'm wrong.

GoPro records the precise start time of a MP4 video in the "tmcd" i.e. GoPro TCD track.
Using the ffmpeg-python package, we can read metadata from the mp4 by:

>>> probe = ffmpeg.probe("GX010114.MP4")
>>> print(probe["streams"][2])
{'index': 2,
 'codec_type': 'data',
 'codec_tag_string': 'tmcd',
 'avg_frame_rate': '60000/1001',
...
 'tags': {'creation_time': '2023-08-03T18:04:42.000000Z',
  'language': 'eng',
  'handler_name': '\x0bGoPro TCD  ',
  'timecode': '18:03:37:22'}}
>>> timecode = probe["streams"][2]['tags']['timecode']
>>> print(timecode)
"18:03:37:22"

This timecode is identical to the one read by ffprobe

  Stream #0:2[0x3](eng): Data: none (tmcd / 0x64636D74), 0 kb/s (default)
    Metadata:
      creation_time   : 2023-08-03T18:04:42.000000Z
      handler_name    : GoPro TCD  
      timecode        : 18:03:37:22

Note that this timecode is recorded in the non-drop frame format, even for 29.97 / 59.94 Hz NTSC videos. Therefore, to convert this timecode into seconds since midnight we need to:

frame_rate = 59.94
int_frame_rate = round(frame_rate) # 60
h, m, s, f = [int(x) for x in timecode.split(':')]
frames = (3600 * h + 60 * m + s) * int_frame_rate + f #3901042
seconds = frames / frame_rate # 65082.384033333336

Alternatively, we can directly read the raw data from tmcd stream, which is a 4 byte big-endian integer for the number of frames since midnight:

fname = "GX010114.MP4"
data = ffmpeg.input(fname)\
            .output("pipe:", format="rawvideo", map="0:2", codec="copy")\
            .run(capture_stdout=True, capture_stderr=True)[0]
tmcd_frames = int.from_bytes(data, byteorder='big') # 3901042
seconds = tmcd_frames / 59.94 # 65082.384033333336

Do NOT use the timecode pip package, since it:

1. assumes drop-frame timecode for 29.97 and 59.94 Hz framerates (but can be forced to non-drop frame)
2. calculates number of frames from seconds using the int_framerate which I believe is incorrect.