Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement VisibilityThresholdModifier #173

Merged
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions Sources/StreamVideo/WebRTC/StreamVideoCaptureHandler.swift
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,10 @@
//

import Foundation
import WebRTC
@preconcurrency import WebRTC

final class StreamVideoCaptureHandler: NSObject, RTCVideoCapturerDelegate {

class StreamVideoCaptureHandler: NSObject, RTCVideoCapturerDelegate {

let source: RTCVideoSource
let filters: [VideoFilter]
let context: CIContext
Expand Down Expand Up @@ -107,3 +107,5 @@ class StreamVideoCaptureHandler: NSObject, RTCVideoCapturerDelegate {
)
}
}

extension StreamVideoCaptureHandler: @unchecked Sendable {}
7 changes: 0 additions & 7 deletions Sources/StreamVideoSwiftUI/CallView/CallView.swift
Original file line number Diff line number Diff line change
Expand Up @@ -159,18 +159,11 @@ public struct CallView<Factory: ViewFactory>: View {
viewFactory.makeVideoParticipantsView(
viewModel: viewModel,
availableSize: size,
onViewRendering: handleViewRendering(_:participant:),
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to do this without breaking changes to the public API?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I remove this one as it's not being used anywhere. We can definitely go ahead without this change but makes our API confusing as we are asking for something what we won't be using.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other things to keep in mind when looking on the screenshots (as you may missed the note below them):

  • The before screenshot shows lower consumption because the app at that point was unresponsive. Nothing was updating on the screen and that was causing consumption to drop.
  • The after screenshot was captured after 55minutes in a call while the before it was on minute 18.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, you should sell it better then 😄

onChangeTrackVisibility: viewModel.changeTrackVisibility(for:isVisible:)
)
}

private var participants: [CallParticipant] {
viewModel.participants
}

private func handleViewRendering(_ view: VideoRenderer, participant: CallParticipant) {
view.handleViewRendering(for: participant) { size, participant in
viewModel.updateTrackSize(size, for: participant)
}
}
}
53 changes: 26 additions & 27 deletions Sources/StreamVideoSwiftUI/CallView/ParticipantsGridView.swift
Original file line number Diff line number Diff line change
Expand Up @@ -8,37 +8,40 @@ import WebRTC

@MainActor
struct ParticipantsGridView<Factory: ViewFactory>: View {

var viewFactory: Factory
var call: Call?
var participants: [CallParticipant]
var availableSize: CGSize
var isPortrait: Bool
var participantVisibilityChanged: (CallParticipant, Bool) -> Void

var body: some View {
ScrollView {
if #available(iOS 14.0, *) {
LazyVGrid(
columns: [
.init(.adaptive(minimum: size.width, maximum: size.width), spacing: 0)
],
spacing: 0
) {
participantsContent
}
.frame(width: availableSize.width)
} else {
VStack {
participantsContent
GeometryReader { geometryProxy in
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like we might have an extra geometry reader? The availableSize value should also come from a geometry reader. Ideally, we should use only one.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

availableSize is indeed coming from a GeometryReader in the CallView. We can't really use it here as we don't need only the size but the origin too.

We can potentially use only one GeometryReader but that will require a change on the API that will allow us to pass availableRect/availableViewport: CGRect instead of availableSize: CGSize.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would say it's better to break the API at this point, since we are still in beta. Geometry readers can be expensive, best to use them only when needed. Wdyt?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changing this one I think will incur many changes. How do you feel about having it in another PR?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe another PR towards this PR, to see how much it is? Since if we do breaking changes, we should do them very soon.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A PR to this branch sounds good. Here it is #176

ScrollView {
if #available(iOS 14.0, *) {
LazyVGrid(
columns: [
.init(.adaptive(minimum: size.width, maximum: size.width), spacing: 0)
],
spacing: 0
) {
participantsContent(geometryProxy.frame(in: .global))
ipavlidakis marked this conversation as resolved.
Show resolved Hide resolved
}
.frame(width: availableSize.width)
} else {
VStack {
participantsContent(geometryProxy.frame(in: .global))
}
}
}
}
.edgesIgnoringSafeArea(.all)
.accessibility(identifier: "gridScrollView")
}

private var participantsContent: some View {

@ViewBuilder
private func participantsContent(_ bounds: CGRect) -> some View {
ForEach(participants) { participant in
viewFactory.makeVideoParticipantView(
participant: participant,
Expand All @@ -57,17 +60,13 @@ struct ParticipantsGridView<Factory: ViewFactory>: View {
showAllInfo: true
)
)
.onAppear {
log.debug("Participant \(participant.name) is visible")
participantVisibilityChanged(participant, true)
}
.onDisappear {
log.debug("Participant \(participant.name) is not visible")
participantVisibilityChanged(participant, false)
.visibilityObservation(in: bounds) {
log.debug("Participant \(participant.name) is \($0 ? "visible" : "not visible.")")
participantVisibilityChanged(participant, $0)
}
}
}

var ratio: CGFloat {
if isPortrait {
let width = availableSize.width / 2
Expand All @@ -79,7 +78,7 @@ struct ParticipantsGridView<Factory: ViewFactory>: View {
return width / height
}
}

private var size: CGSize {
if #available(iOS 14.0, *) {
let dividerWidth: CGFloat = isPortrait ? 2 : 3
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -53,39 +53,32 @@ public struct ParticipantsSpotlightLayout<Factory: ViewFactory>: View {
showAllInfo: true
)
)
.onAppear {
log.debug("Participant \(participant.name) is visible")
onChangeTrackVisibility(participant, true)
}
.modifier(ParticipantChangeModifier(
participant: participant,
onChangeTrackVisibility: onChangeTrackVisibility)
)

ScrollView(.horizontal) {
HorizontalContainer {
ForEach(participants) { participant in
viewFactory.makeVideoParticipantView(
participant: participant,
id: participant.id,
availableSize: .init(width: thumbnailSize, height: thumbnailSize),
contentMode: .scaleAspectFill,
customData: [:],
call: call
)
.onAppear {
onChangeTrackVisibility(participant, true)
}
.onDisappear {
onChangeTrackVisibility(participant, false)
GeometryReader { geometry in
HorizontalContainer {
ForEach(participants) { participant in
viewFactory.makeVideoParticipantView(
participant: participant,
id: participant.id,
availableSize: .init(width: thumbnailSize, height: thumbnailSize),
contentMode: .scaleAspectFill,
customData: [:],
call: call
)
.visibilityObservation(in: geometry.frame(in: .global)) { onChangeTrackVisibility(participant, $0) }
.adjustVideoFrame(to: thumbnailSize, ratio: 1)
.cornerRadius(8)
.accessibility(identifier: "spotlightParticipantView")
}
.adjustVideoFrame(to: thumbnailSize, ratio: 1)
.cornerRadius(8)
.accessibility(identifier: "spotlightParticipantView")
}
.frame(height: thumbnailSize)
.cornerRadius(8)
}
.frame(height: thumbnailSize)
.cornerRadius(8)
}
.padding()
.padding(.bottom)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,6 @@ public struct VideoParticipantsView<Factory: ViewFactory>: View {
var viewFactory: Factory
@ObservedObject var viewModel: CallViewModel
var availableSize: CGSize
var onViewRendering: (VideoRenderer, CallParticipant) -> Void
var onChangeTrackVisibility: @MainActor(CallParticipant, Bool) -> Void

@State private var orientation = UIApplication.shared.windows.first?.windowScene?.interfaceOrientation ?? .unknown
Expand All @@ -20,13 +19,11 @@ public struct VideoParticipantsView<Factory: ViewFactory>: View {
viewFactory: Factory,
viewModel: CallViewModel,
availableSize: CGSize,
onViewRendering: @escaping (VideoRenderer, CallParticipant) -> Void,
onChangeTrackVisibility: @escaping @MainActor(CallParticipant, Bool) -> Void
) {
self.viewFactory = viewFactory
self.viewModel = viewModel
self.availableSize = availableSize
self.onViewRendering = onViewRendering
self.onChangeTrackVisibility = onChangeTrackVisibility
}

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
//
// Copyright © 2023 Stream.io Inc. All rights reserved.
//

import StreamVideo
import SwiftUI

/// The modifier designed to dynamically track and respond to the visibility status of a view within its parent
/// bounds or viewport. It utilises a user-defined visibility threshold, represented as a percentage, to
/// determine how much of the view should be visible (both vertically and horizontally) before it's considered
/// "on screen".
///
/// When the visibility state of the view changes (i.e., it transitions between being "on screen" and "off screen"),
/// a callback is triggered to notify the user of this change. This can be particularly useful in scenarios where
/// resource management is crucial, such as video playback or dynamic content loading, where actions might
/// be triggered based on whether a view is currently visible to the user.
///
/// By default, the threshold is set to 30%, meaning 30% of the view's dimensions must be within the parent's
/// bounds for it to be considered visible.
struct VisibilityThresholdModifier: ViewModifier {
/// State to track if the content view is on screen.
@State private var isOnScreen = false {
didSet {
// Check if the visibility state has changed.
guard isOnScreen != oldValue else { return }
// Notify the caller about the visibility state change.
changeHandler(isOnScreen)
}
}

/// The bounds of the parent view or viewport.
var bounds: CGRect
/// The threshold percentage of the view that must be visible.
var threshold: CGFloat
/// Closure to handle visibility changes.
var changeHandler: (Bool) -> Void

init(in bounds: CGRect,
threshold: CGFloat,
changeHandler: @escaping (Bool) -> Void
) {
self.bounds = bounds
self.threshold = threshold
self.changeHandler = changeHandler
}

func body(content: Content) -> some View {
content
.background(
GeometryReader { geometry -> Color in
/// Convert the local frame of the content to a global frame.
let geometryInGlobal = geometry.frame(in: .global)

let (verticalVisible, horizontalVisible) = calculateVisibilityInBothAxis(in: geometryInGlobal)

/// Update the isOnScreen state based on visibility calculations.
DispatchQueue.main.async {
self.isOnScreen = verticalVisible && horizontalVisible
}

/// Use a clear color for the background to not affect the appearance.
return Color.clear
}
)
}

func calculateVisibilityInBothAxis(in rect: CGRect) -> (verticalVisible: Bool, horizontalVisible: Bool) {
/// Calculate the global minY, maxY, minX, and maxX of the content view.
let minY = rect.minY
let maxY = rect.maxY
let minX = rect.minX
let maxX = rect.maxX

/// Calculate required height and width based on visibility threshold.
let requiredHeight = rect.size.height * threshold
let requiredWidth = rect.size.width * threshold

/// Check if the content view is vertically within the parent's bounds.
let verticalVisible = (minY + requiredHeight < bounds.maxY && minY > bounds.minY) ||
(maxY - requiredHeight > bounds.minY && maxY < bounds.maxY)
/// Check if the content view is horizontally within the parent's bounds.
let horizontalVisible = (minX + requiredWidth < bounds.maxX && minX > bounds.minX) ||
(maxX - requiredWidth > bounds.minX && maxX < bounds.maxX)

return (verticalVisible, horizontalVisible)
}
}

extension View {
/// Attaches a visibility observation modifier to the view.
///
/// - Parameters:
/// - bounds: The bounds of the parent view or viewport within which the visibility of the view will
/// be tracked.
/// - threshold: A percentage value (defaulted to 0.3 or 30%) representing how much of the view
/// should be visible within the `bounds` before it's considered "on screen".
/// - changeHandler: A closure that gets triggered with a Boolean value indicating the visibility
/// state of the view whenever it changes.
///
/// - Returns: A modified view that observes its visibility status within the specified bounds.
func visibilityObservation(
in bounds: CGRect,
threshold: CGFloat = 0.3,
changeHandler: @escaping (Bool) -> Void
) -> some View {
modifier(
VisibilityThresholdModifier(
in: bounds,
threshold: threshold,
changeHandler: changeHandler
)
)
}
}
6 changes: 1 addition & 5 deletions Sources/StreamVideoSwiftUI/ViewFactory.swift
Original file line number Diff line number Diff line change
Expand Up @@ -43,13 +43,11 @@ public protocol ViewFactory: AnyObject {
/// - Parameters:
/// - viewModel: The view model used for the call.
/// - availableSize: the size available for rendering.
/// - onViewRendering: called when the video view is rendered.
/// - onChangeTrackVisibility: called when a track changes its visibility.
/// - Returns: view shown in the video participants slot.
func makeVideoParticipantsView(
viewModel: CallViewModel,
availableSize: CGSize,
onViewRendering: @escaping (VideoRenderer, CallParticipant) -> Void,
ipavlidakis marked this conversation as resolved.
Show resolved Hide resolved
onChangeTrackVisibility: @escaping @MainActor(CallParticipant, Bool) -> Void
) -> ParticipantsViewType

Expand Down Expand Up @@ -195,14 +193,12 @@ extension ViewFactory {
public func makeVideoParticipantsView(
viewModel: CallViewModel,
availableSize: CGSize,
onViewRendering: @escaping (VideoRenderer, CallParticipant) -> Void,
onChangeTrackVisibility: @escaping @MainActor(CallParticipant, Bool) -> Void
) -> some View {
VideoParticipantsView(
viewFactory: self,
viewModel: viewModel,
availableSize: availableSize,
onViewRendering: onViewRendering,
onChangeTrackVisibility: onChangeTrackVisibility
)
}
Expand Down Expand Up @@ -333,7 +329,7 @@ extension ViewFactory {
}

public class DefaultViewFactory: ViewFactory {

private init() { /* Private init. */ }

public static let shared = DefaultViewFactory()
Expand Down
Loading
Loading