Add: Latency measurements such as sender to receiver latency measurements using RTP header extensions abs-capture-time and video-timing #425

lukehb · 2025-01-30T01:15:16Z

Relevant components:

Common library
Frontend library
Frontend UI library

Problem statement:

Adds optional abs-capture-time RTP header extension to the negotiated SDP (only works with Chromium+audio as of now). Additionally use the video-timing header extension (only works in Chromium+video as of now) to get video statistics from both sender and receiver time including: encoding delay, packetizer delay, pacer delay.

Solution

Adds a new config flag to enable abs-capture-time to RTP header extension
Munges the extension into the SDP
Adds appropriate tests to use this header extension to measure latency using captureTimestamp.
Adds some more latency stats to the frontend stats panel to display the moment to moment latency using information from both abs-capture-time (if possible) and video-timing.

Note: these only show up in supported browsers - that is, much of this does nothing on Firefox because many of these code paths are completely skipped on Firefox.

Documentation

N/A - The flag is enabled by default and not exposed to the UI. It should be a transparent change to user and should not need configuration.

Test Plan and Compatibility

All unit tests, playwright tests, and CI have been updated accordingly (this was tricky as testing revealed Firefox refuses to connect if this flag there are too many header extensions so this is carefully skipped on Firefox).

…is enabled.

…ld by only installing relevant playwright browsers.

… before starting the stream

… test streamer

lukehb · 2025-01-30T04:21:28Z

Based on this: https://github.com/EpicGamesExt/PixelStreamingInfrastructure/actions/runs/13026995654

@mcottontensor Landing this PR would require new common and lib-pixelstreamingfrontend versions to be cut.

lukehb · 2025-01-30T04:23:18Z

.github/workflows/healthcheck-libraries.yml

+      run: npm install && npm run build
+
+    - name: Clean build of frontend implementation as ES6


Added this step to the CI, as this was actually broken and not being caught.

lukehb · 2025-01-30T04:23:48Z

.github/workflows/healthcheck-streaming.yml

-        tag: 'minimal-streamer'
-        fileName: 'Minimal-PixelStreamer-5.5.7z'
+        tag: 'minimal-streamer-5.5'
+        fileName: 'Minimal-PixelStreamer-5.5-Win64-Development.7z'


Upgraded to using our latest minimal streamer release, as the other one was a bit out of date now.

Extras/FrontendTests/playwright.config.ts

Extras/FrontendTests/tests/fixtures.ts

Extras/FrontendTests/tests/keyboard.spec.ts

Extras/JSStreamer/Dockerfile

.github/workflows/healthcheck-streaming.yml

Extras/FrontendTests/playwright.config.ts

Extras/FrontendTests/tests/helpers.ts

.github/workflows/healthcheck-streaming.yml

Extras/FrontendTests/tests/helpers.ts

Frontend/implementations/typescript/tsconfig.esm.json

lukehb · 2025-02-07T06:04:10Z

~~Note to self: I need to merge changes from master into here to fix CI and additionally I need to test this change on more browser and devices (iOS / Android / Safari / FF).~~ Done!

lukehb · 2025-02-10T00:14:17Z

Merged master in now.

~~Still need to test on devices/browser~~ Done! All good.

Also want to make which capture to send method is used clearer. This is because both video-timing and abs-capture-time both can calculate it, but it is unclear to me if these are the same/similar due to Chrome not yet supporting remote-outbound-rtp stats for video. Because Chrome does not support these stats, I can't calculate capture to send for video using abs-capture-time. Therefore, the current commit falls back to using video timing to calculate capture to send time, but we should make this clearer this approach is used. Done: ee5d7bd

lukehb · 2025-02-12T05:28:42Z

@mcottontensor When we go to cut a new version from this, there is some API breakages due to some changing of stats casing. So I think we have bump the major version number.

mcottontensor · 2025-02-13T00:41:28Z

Do we want to do a full release? Or just common and the frontend libs?

github-actions · 2025-02-13T00:46:25Z

💔 All backports failed

Status	Branch	Result
❌	UE5.5	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

backport --pr 425

Questions ?

Please refer to the Backport tool documentation and see the Github Action logs for details

Add: Latency measurements such as sender to receiver latency measurements using RTP header extensions abs-capture-time and video-timing (cherry picked from commit 063ca4f)

mcottontensor · 2025-02-13T03:29:06Z

💚 All backports created successfully

Status	Branch	Result
✅	UE5.5

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

[UE5.5] Merge pull request #425 from EpicGamesExt/capture-time

rlamarche · 2025-03-02T16:26:51Z

Frontend/library/src/Config/Config.ts

+                'Enables the abs-capture-time RTP header extension',
+                settings && Object.prototype.hasOwnProperty.call(settings, Flags.EnableCaptureTimeExt)
+                    ? settings[Flags.EnableCaptureTimeExt]
+                    : true,


Hi, is it voluntary to enable this by default ? It breaks compatiblity with not up to date Chrome/Chromium browser.

@rlamarche Thanks for reporting, I have put together a PR: #516

lukehb added 23 commits December 6, 2024 17:38

Add sending abs-capture-time in JS streamer and a test

b9e12d2

Add some comments

fc83cee

Added extra commenting

b950af3

Added SDPUtils to /Common

47398a8

Working state

4a3d131

Added test to check if abs-capture-time is in offer/answer when flag …

cfcbdf9

…is enabled.

Update JSStreamer dockerfile to build using local files. Speed up bui…

cf1cf95

…ld by only installing relevant playwright browsers.

Added SDP offer/remote description set event to PixelStreaming API

911ead6

Connect new latency calculation API to stats panel

1f5b9f8

Added latency info to tests, fixed linting

e15961a

Fix failing unit test

56ae963

Fix linting

8ef4748

Attempt to fix test race condition through explicitly adding listener…

6e85d66

… before starting the stream

Attempt to update playwright version used in minimal stream tester

47f041c

Remove problematic test teardown when test was skipped

514089c

Add extra info step to streaming CI job to help diagnose CI failures

2c47b94

Iterating on what to log

ee4edc2

Attempt to revert some load bearing changes and add logging to stream…

4f1ea36

… test streamer

Move to using latest minimal streamer release

df82e74

Disable header extension for Firefox as it causes it to not connect

c5c46e0

Made code more exclusive of Firefox weirdness

4b4db38

Linting fix

1833e62

Merge branch 'master' into capture-time

1e707f5

lukehb requested a review from mcottontensor January 30, 2025 01:15

lukehb added auto-backport Used to specify we want a PR to auto backport to a branch, must be paired with auto-backport-to-UEX. auto-backport-to-UE5.5 labels Jan 30, 2025

lukehb commented Jan 30, 2025

View reviewed changes

Extras/FrontendTests/playwright.config.ts Show resolved Hide resolved

lukehb commented Jan 30, 2025

View reviewed changes

Extras/FrontendTests/tests/fixtures.ts Show resolved Hide resolved

lukehb commented Jan 30, 2025

View reviewed changes

Extras/FrontendTests/tests/keyboard.spec.ts Show resolved Hide resolved

lukehb commented Jan 30, 2025

View reviewed changes

Extras/JSStreamer/Dockerfile Show resolved Hide resolved

mcottontensor reviewed Jan 30, 2025

View reviewed changes

.github/workflows/healthcheck-streaming.yml Show resolved Hide resolved

.github/workflows/healthcheck-streaming.yml Show resolved Hide resolved

Extras/FrontendTests/playwright.config.ts Show resolved Hide resolved

Extras/FrontendTests/tests/helpers.ts Outdated Show resolved Hide resolved

lukehb added 2 commits January 31, 2025 16:37

Fixing linting error

989a62a

Tided up how latency stats are displayed in stats panel

701c265

lukehb commented Feb 7, 2025

View reviewed changes

.github/workflows/healthcheck-streaming.yml Outdated Show resolved Hide resolved

lukehb commented Feb 7, 2025

View reviewed changes

.github/workflows/healthcheck-streaming.yml Outdated Show resolved Hide resolved

lukehb commented Feb 7, 2025

View reviewed changes

Extras/FrontendTests/tests/helpers.ts Outdated Show resolved Hide resolved

lukehb commented Feb 7, 2025

View reviewed changes

Frontend/implementations/typescript/tsconfig.esm.json Outdated Show resolved Hide resolved

Apply suggestions from code review

e00a2e5

Merge branch 'master' into capture-time

0b48276

lukehb changed the title ~~Add: Optional abs-capture-time RTP header extension to enable browser-side sender to receiver latency measurements~~ Add: Latency measurements such as sender to receiver latency measurements using RTP header extensions abs-capture-time and video-timing Feb 10, 2025

lukehb added 3 commits February 12, 2025 16:33

Merge branch 'master' into capture-time

3314c9c

Fix failing CI for build:esm job

070bbf8

Made it clearer whether video timing or abs-capture-time is used

ee5d7bd

mcottontensor approved these changes Feb 12, 2025

View reviewed changes

mcottontensor merged commit 063ca4f into master Feb 13, 2025
9 checks passed

mcottontensor deleted the capture-time branch February 13, 2025 00:46

mcottontensor mentioned this pull request Feb 13, 2025

[UE5.5] Merge pull request #425 from EpicGamesExt/capture-time #449

Merged

mcottontensor added a commit that referenced this pull request Feb 13, 2025

Merge pull request #449 from mcottontensor/backport/UE5.5/pr-425

c693277

[UE5.5] Merge pull request #425 from EpicGamesExt/capture-time

rlamarche reviewed Mar 2, 2025

View reviewed changes

lukehb mentioned this pull request Mar 2, 2025

Make abs-capture-time flag false by default #516

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add: Latency measurements such as sender to receiver latency measurements using RTP header extensions abs-capture-time and video-timing #425

Add: Latency measurements such as sender to receiver latency measurements using RTP header extensions abs-capture-time and video-timing #425

lukehb commented Jan 30, 2025 •

edited

Loading

lukehb commented Jan 30, 2025

lukehb Jan 30, 2025

lukehb Jan 30, 2025 •

edited

Loading

lukehb commented Feb 7, 2025 •

edited

Loading

lukehb commented Feb 10, 2025 •

edited

Loading

lukehb commented Feb 12, 2025

mcottontensor commented Feb 13, 2025

github-actions bot commented Feb 13, 2025

mcottontensor commented Feb 13, 2025

rlamarche Mar 2, 2025

lukehb Mar 2, 2025

		run: npm install && npm run build

		- name: Clean build of frontend implementation as ES6

Add: Latency measurements such as sender to receiver latency measurements using RTP header extensions abs-capture-time and video-timing #425

Add: Latency measurements such as sender to receiver latency measurements using RTP header extensions abs-capture-time and video-timing #425

Conversation

lukehb commented Jan 30, 2025 • edited Loading

Relevant components:

Problem statement:

Solution

Documentation

Test Plan and Compatibility

lukehb commented Jan 30, 2025

lukehb Jan 30, 2025

Choose a reason for hiding this comment

lukehb Jan 30, 2025 • edited Loading

Choose a reason for hiding this comment

lukehb commented Feb 7, 2025 • edited Loading

lukehb commented Feb 10, 2025 • edited Loading

lukehb commented Feb 12, 2025

mcottontensor commented Feb 13, 2025

github-actions bot commented Feb 13, 2025

💔 All backports failed

Manual backport

Questions ?

mcottontensor commented Feb 13, 2025

💚 All backports created successfully

Questions ?

rlamarche Mar 2, 2025

Choose a reason for hiding this comment

lukehb Mar 2, 2025

Choose a reason for hiding this comment

lukehb commented Jan 30, 2025 •

edited

Loading

lukehb Jan 30, 2025 •

edited

Loading

lukehb commented Feb 7, 2025 •

edited

Loading

lukehb commented Feb 10, 2025 •

edited

Loading