what audio-only tests are needed?

[jpiesing]

What audio-only tests are needed?

For video, I created a sparse-matrix identifying which of the HTML+JavaScript test code (templates) needed running with which of the test content. This helped confirm that we had the correct test content. It was used to create a .csv file which generated the actual tests.

This issue is to track the creation of that .csv file for the audio-only tests, e.g. the Dolby ones whose test content is being developed by Eurofins.

[cta-source]

From requirements analysis of the DPCTF spec as presented in DPCTF 9/1/2021:

• DPC Specification Audio Observation Requirements (sections 8 & 9):

1. Every audio sample renders
2. Audio samples render in increasing presentation time order
3. Audio playback duration matches expected value, at audio-sample-level resolution.
4. For random access (time or fragment), no earlier audio sample shall be rendered
5. Audio startup delay to be sufficiently low, at audio-sample-level resolution.
6. The presented audio sample matches the one reported by currentTime.

These requirements lead to:
Audio Startup Delay test" -- Verifies that audio playout begins within a specified minimum time. Given audio recorded from device-under-test playout, establishes presentationTime T0=0 from video frame markers, and T=Tstart (relative to the embedded mezzanine watermark), then verifies that this T=Tstart is within the startup delay limits in the specification. [Edit: Is this the right way to phrase this, specifically the "presentationTime from video frame markers" part?] (Meets requirement 5, above.)
Audio Segment Playout test" -- Verifies that each audio sample renders, in the correct order and increasing presentation time. Given audio recorded from device-under-test playout, establishes a time T0=0 relative to start of its embedded mezzanine watermark, then verifies that the watermark for each 20mS segment of the recorded audio shows up in the recorded audio at the expected delay from T0=0. (Meets requirements 1-3, above.)
Random Access test -- Given recorded audio that includes a known random access (time or fragment), and given the expected presentation time of first N 20mS segments after the random access, verify these segments are presented as expected, in increasing order and without gaps. (Meets requirement 4, above.)
TBD currentTime Test -- How to design this? (To meet requirement 6, above.)

Also, from discussion in Mezzanine/Issue 37, additional utility tests:

System SNR Estimation test -- Automated test; use to verify equipment meets basic minimum quality-of-transmission requirements
Chord Progressions test -- Human listening test; use when desiring to hear basic system operation. Built from synthesized chords with a click before and after each one, e.g.: Click, C-major (1 second), click, D-major (1s), click, E-major (1s), etc. Actual chords are tbd.

A final note, there are TBDs in the DPCTF spec regarding some audio observations. As these are resolved into actual requirements, new tests may be required.

[jpiesing]

@cta-source I may be missing something but the list above appears to start from a clean sheet whereas I was expecting we would be re-using the template HTML+JS from the video-only tests. Look at sections 8 and 9 of the DPCTF spec.

I was expecting something like this;
8.2 - run once for each option for how an audio codec bitstream can be packaged into ISOBMFF (options within the bitstream are assumed to be tested by the bitstream test suite)
8.3, 8.4, 8.6, 8.7, 8.9, 9.2, 9.3 - run once with one CMAF track as input
8.5 - run once but input is a CMAF switching set, so needs several different audio tracks
8.8, 9.4 - run once but input is 2 CMAF switching sets, each with one audio track so needs different audio tracks
8.10 - run once but input is a CMAF switching set with multiple tracks - perhaps the same as 8.5 (tbc)?
8.11 - not relevant for audio codecs
8.12 - run once with an encrypted audio track as input
8.13 - don't understand. Perhaps same as 8.8?
8.14 - run once, same as 8.8 but with one of the switching sets encrypted

[cta-source]

I also assume that we will re-use the template HTML+JS. I'm not sure, but I think we're at different points in implementation--my list is still an intermediary step. I'm mapping the tests I identified to 8.2, 8.3, etc. So rather than craft a unique test for each item, there will eventually be buried into the test suite certain tests or functions that need to be called for each case.

So 8.2 requires the following:

1. Audio Segment Playout test (a version is being refined now)
2. Audio Startup Delay test (a version is being refined now, same as Audio Segment Playout)
3. currentTime test (to be defined)

@gitwjr and I have been discussing and documenting this mapping.

However, eventually someone needs to do the integration that would presumably re-use the template HTML+JSON.

[jpiesing]

It may be that I'm missing something about what information is being embedded in the audio & with what granularity.

For video, each frame includes stream identifier, timecode of the frame, frame number, frame rate.
What is included in each 20ms audio segment?
What bitrate does this technique give?

[cta-source]

Short answer: No encoded data other than a long always-the-same PN string, from which we can extract timing but nothing else. Bitrate is effectively 0 after you extract timing information. I'm assuming we get all that data (stream identifier, etc.) from the video portion of the test, and the audio portion of the test is just verifying the audio playout characteristics.

(This "timing-only" limitation is obviously a problem for audio-only stream testing, should that become a priority. See below for options in that case.)

Long answer (get coffee now):
The best way to think of the PN sequence is as a blank data carrier, not as actual data. To send actual data, there are two steps: establish a carrier, then add your data via carrier modulation. For this PN technique, the first step (establish a carrier) is to embed the PN sequence. That's a 2,880,000 bit length string of 1's and 0's with no other meaning, just the appearance of randomness.

From this always-the-same string, we can extract timing information but nothing else. That's what we're doing so far.

After you have timing information, you can do more (which we are not yet doing):

To encode useful data, we need a modulation scheme. Here I would take some subframe of the full 2.8M bits--say, 1000 bits at a time--and do something to it that you can work out later, like invert all of a subframe's individual 1's and 0's to encode a '0', but leave them the same for '1'. That would allow you to take the 2,880,000 bits of PN sequence and have 2,880 bits of encoded data.

The penalty is that the 1000:1 loss of data rate is exactly 30dB of SNR penalty in demodulating (for time in the first part, and useful data in the second part). We can no longer use a full 2,880,000 string of data to check the start of timing, we've broken it up into 1000 smaller pieces and have to look for each piece separately. As you know, we're already worried about SNR issues for some test environment scenarios.

However, we're already incurring that penalty because we need 20mS resolution on timing--which means we check about 1000 bits at a time anyway (actually, 960 bits, but close enough).

We could encode one bit per 20mS sequence eventually, with (I think) about a 2:1 penalty on test time. We'd get an effective rate of 50bps encoded 'raw' data. In 60 seconds that would be roughly 150-200 effective bytes (assuming usual overhead for this sort of data transmission, putting the raw data into packets, checksums, HDLC bit stuffing, etc.)

This is a theory, I've never done it this way (inverting segments of a PN sequence). I can think of a reason it might not work, but it might be fine. Happy to put it into the queue to try it someday, but I'd like to continue to focus on the current goals, ofc.
[Edit: changed "now" to "not"]

[yanj-github]

@cta-source I may be missing something but the list above appears to start from a clean sheet whereas I was expecting we would be re-using the template HTML+JS from the video-only tests. Look at sections 8 and 9 of the DPCTF spec.

I was expecting something like this;
8.2 - run once for each option for how an audio codec bitstream can be packaged into ISOBMFF (options within the bitstream are assumed to be tested by the bitstream test suite)
8.3, 8.4, 8.6, 8.7, 8.9, 9.2, 9.3 - run once with one CMAF track as input
8.5 - run once but input is a CMAF switching set, so needs several different audio tracks
8.8, 9.4 - run once but input is 2 CMAF switching sets, each with one audio track so needs different audio tracks
8.10 - run once but input is a CMAF switching set with multiple tracks - perhaps the same as 8.5 (tbc)?
8.11 - not relevant for audio codecs
8.12 - run once with an encrypted audio track as input
8.13 - don't understand. Perhaps same as 8.8?
8.14 - run once, same as 8.8 but with one of the switching sets encrypted

Thanks @jpiesing this is inline with what we are thinking.
In addition to this, I guess we need to define/choose which video mpd goes with each of these audio tests. Probabaly we can pick same video mpd for different audios? What do you think please?
The reason to add video mpd for audio test is:

• Test Runner seems only plays back content when video mpd is defined
• Some device doesnt support audio only playback
• Observation such as start up delay check and current time check requires Test Runner QR code which I am not sure if it will display without video

[louaybassbouss]

• Test Runner seems only plays back content when video mpd is defined

@yanj-github for now yes we can play video only or video+audio tests. For audio only tests a minor change in the core player is needed but easy to support audio only tests from our side. In case the group decide for audio only tests, this is not a blocker.

• Observation such as start up delay check and current time check requires Test Runner QR code which I am not sure if it will display without video

All QR codes produced by the test runner will be displayed independent if the underlying <video> element plays audio only or audio+video.

[louaybassbouss]

PS: audio-only tests are now supported in the feature-multi-mpd branch which will be merged into master soon after @yanj-github confirms

[yanj-github]

Just to add, splicing and switching tests are not included in e-ac-3 and ac-4 tests that we have generated. These are still under discussion.

[yanj-github]

@louaybassbouss and @FritzHeiden we are about to get a new OF realese out soon by the end of this week.
Can you kindly merge feature-multi-mpd branch please?

[louaybassbouss]

@yanj-github the feature-multi-mpd is now merged into the master branch can you please check.

[yanj-github]

Thanks @louaybassbouss it looks good to me.

[louaybassbouss]

@yanj-github I think we can close this issue can you confirm?

[jpiesing]

The original subject of this issue has been moved to cta-wave/Test-Content#24 so I think what's left can be closed.

[yanj-github]

I think this one is fine to be closed a well.