Skip to content

[camera_avfoundation] handle interruptions and use single offset #8982

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open

[camera_avfoundation] handle interruptions and use single offset #8982

wants to merge 7 commits into from

Conversation

misos1
Copy link
Contributor

@misos1 misos1 commented Apr 2, 2025

Handle interruptions of capture sessions by doing "internal pause". Another possibility would be to remove audio input from the session but this would be too specific for interruptions caused by an incoming call or alarm. Or to stop video recording as the native ios camera app does but this would require some additional event.

Handle asynchronous error notifications posted about capture sessions. This can happen for example when there is an incoming phone call and the camera starts recording with audio. For this specific case would be enough to just call startRunning on capture sessions but that is too specific and would require ability to restart capture session from dart side as seems there is no good way how to test when such phone call or other such event ends, and trying to restart it for example every second inside FLTCam does not look like good idea. Instead the camera controller enters an error state and can be recovered by disposing and recreating which should be more generic.

Use single offset for both video and audio as separate offsets can cause unsynchronized video with audio. Seems audio timestamps tend to be ignored except for the first sample so prefer them for calculating offset to avoid desynchronization. Avoid errors when trying to append samples with timestamp less or equal to the previous sample. Fix a bug when after adjusting offset also last sample time needs to be updated to avoid adding past offsets multiple times.

Pre-Review Checklist

If you need help, consider asking for advice on the #hackers-new channel on Discord.

Footnotes

  1. Regular contributors who have demonstrated familiarity with the repository guidelines only need to comment if the PR is not auto-exempted by repo tooling. ↩ ↩2 ↩3

@misos1
Copy link
Contributor Author

misos1 commented Apr 3, 2025 •
edited
Loading 

Mac_arm64 ios_platform_tests_shard_2 master
Testing failed:
	Type 'AVCaptureSession' has no member 'wasInterruptedNotification'
	Command SwiftCompile failed with a nonzero exit code
	Testing cancelled because the build failed.

I am not sure what this should mean. AVCaptureSession.wasInterruptedNotification should be there since ios 4 (I managed to run tests successfully locally on real device and also on simulator).

hellohuanlin
hellohuanlin reviewed Apr 29, 2025
View reviewed changes
packages/camera/camera_avfoundation/example/ios/RunnerTests/CameraTestUtils.swift
@@ -80,7 +80,9 @@ enum CameraTestUtils {

/// Creates a test sample buffer.
/// @return a test sample buffer.
static func createTestSampleBuffer() -> CMSampleBuffer {
static func createTestSampleBuffer(
timestamp: CMTime = CMTime.zero, duration: CMTime = CMTimeMake(value: 1, timescale: 44100)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: can infer .zero
nit2: split into multiple lines

packages/camera/camera_avfoundation/CHANGELOG.md
## 0.9.18+14

* Handle video and audio interruptions and errors.
* Use a single time offset for both video and audio.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

... to fix a bug where audio and video are out of sync after interrupted by phone cals.

...ages/camera/camera_avfoundation/ios/camera_avfoundation/Sources/camera_avfoundation/FLTCam.m

[NSNotificationCenter.defaultCenter addObserver:self
selector:@selector(captureSessionRuntimeError:)
name:AVCaptureSessionRuntimeErrorNotification
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when will run time error happen?

Copy link
Contributor Author

@misos1 misos1 Apr 29, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As is in the description "for example when there is an incoming phone call and the camera starts recording with audio". When startVideoRecording is called with enabled audio while there is an incoming call (does not need to be answered in my case, ringing is enough). Audio session then stops running so no more audio can be recorded after this happens until the session is started again or FLTCam recreated (seems the only possibility to recover from the dart side for now). Seems no reasonable detection of this state is possible from the dart side for now (sending error to dart side on this error notification enables such detection).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you put some comments there?

...ages/camera/camera_avfoundation/ios/camera_avfoundation/Sources/camera_avfoundation/FLTCam.m
@@ -788,10 +803,8 @@ - (void)newVideoSample:(CMSampleBufferRef)sampleBuffer {
}
return;
}
if (_videoWriterInput.readyForMoreMediaData) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is moved to top right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes.

...ages/camera/camera_avfoundation/ios/camera_avfoundation/Sources/camera_avfoundation/FLTCam.m
@property(assign, nonatomic) CMTime recordingTimeOffset;
@property(assign, nonatomic) CMTime lastSampleEndTime;
@property(nonatomic) AVCaptureOutput *outputForOffsetAdjusting;
@property(assign, nonatomic) CMTime lastAppendedVideoSampleTime;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is lastAppendedVideoSampleTime the same as lastSampleEndTime?

Can you add some doc on these 4 properties?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lastAppendedVideoSampleTime is related to this text from the description "Avoid errors when trying to append samples with timestamp less or equal to the previous sample.". appendPixelBuffer throws an error if a sample with specified sample time (PresentationTime) is appended after another sample with higher or equal time was already appended before (and maybe this should be end time rather than start time too).

...ages/camera/camera_avfoundation/ios/camera_avfoundation/Sources/camera_avfoundation/FLTCam.m
@property(assign, nonatomic) CMTime videoTimeOffset;
@property(assign, nonatomic) CMTime audioTimeOffset;
@property(assign, nonatomic) CMTime recordingTimeOffset;
@property(assign, nonatomic) CMTime lastSampleEndTime;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

am i reading it right that

(1) recordingTimeOffset covers videoTimeOffset and audioTimeOffset and

(2) lastSampleEndTime covers lastVideoSampleTime and lastAudioSampleTime?

Copy link
Contributor Author

@misos1 misos1 Apr 29, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, lastAudioSampleTime was actually the end time of a sample so I named it accordingly. And lastVideoSampleTime was practically end time too, according to logic for offset calculation both before and after this PR, it just happens that video samples tend to not have duration which can be interpreted as zero duration in this case so end time is same as start time. For that logic to work right this needs to be end time of sample, not start time.

...ages/camera/camera_avfoundation/ios/camera_avfoundation/Sources/camera_avfoundation/FLTCam.m
@property(assign, nonatomic) CMTime audioTimeOffset;
@property(assign, nonatomic) CMTime recordingTimeOffset;
@property(assign, nonatomic) CMTime lastSampleEndTime;
@property(nonatomic) AVCaptureOutput *outputForOffsetAdjusting;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we didn't have this output. can this be just a local variable?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That output is either audio or video output depending on whether audio or video output should be used for offset adjusting.

hellohuanlin
hellohuanlin reviewed Apr 29, 2025
View reviewed changes
...ages/camera/camera_avfoundation/ios/camera_avfoundation/Sources/camera_avfoundation/FLTCam.m
@property(assign, nonatomic) CMTime recordingTimeOffset;
@property(assign, nonatomic) CMTime lastSampleEndTime;
@property(nonatomic) AVCaptureOutput *outputForOffsetAdjusting;
@property(assign, nonatomic) CMTime lastAppendedVideoSampleTime;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is lastAppendedVideoSampleTime the same as lastSampleEndTime?

Can you add some doc on these 4 properties?

@stuartmorgan-g
Copy link
Contributor

From triage: what's the status of this PR? It's not clear to me if it's waiting for updates, or waiting for re-review.

@misos1
Copy link
Contributor Author

misos1 commented Jun 6, 2025

It is waiting for updates.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hellohuanlin hellohuanlin hellohuanlin left review comments

@bparrishMines bparrishMines Awaiting requested review from bparrishMines bparrishMines is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
p: camera platform-ios platform-macos
Projects
None yet
Milestone
No milestone
3 participants
@misos1 @stuartmorgan-g @hellohuanlin