-
-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: ✨ V3 ✨ #1466
feat: ✨ V3 ✨ #1466
Conversation
* chore: Upgrade Example to RN 0.71 * chore: Upgrade all libs * fix: Fix CameraRoll installation * Update Gradle Tools * fix: Fix buildscripts * Clean out build.gradle * fix: Fix Kotlin setup * fix: Move kotlin-android dependency to lib * Move `_setGlobalConsole` * Update gradle-wrapper.properties * Rebuild lockfiles * chore: Update build:gradle * Update StatusBarBlurBackground.tsx * Use Java 11 in Workflows * Update MediaPage.tsx * Add `google` repository to build.gradle * Double Java Heap size * Increase heap size * Alternative args * Update build.gradle
This comment was marked as off-topic.
This comment was marked as off-topic.
* Setup RN Worklets * Use RN Worklets on iOS * Fix console * Add `installFrameProcessorBindings()` function * Add `FrameProcessorPlugins` proxy (BREAKING CHANGE) * Clean up docs * Update FRAME_PROCESSORS.mdx * Use RN Worklets 0.2.5 * feat: Android build setup * Rewrite Android Frame Processor Part * Update CMakeLists.txt * fix: Add react-native-worklets Gradle dependency * Update Podfile.lock * fix build * gradle:7.4.1 * Init JSI Bindings in method on Android * Fix Folly flags * fix: Init `FrameProcessorRuntimeManager` later * fix: Wrap in `<GestureHandlerRootView>` * Refactor plugins * fix: Remove enableFrameProcessors * Install RN Worklets from current GH master * Update babel.config.js * Update CameraViewModule.kt * Update ImageProxyUtils.java * feat: Upgrade to Reanimated v3 * fix: Fix crash on Worklet init * Update RN Worklets to latest master * fix: Simplify FP Plugins Proxy
…1472) Before, Frame Processors ran on a separate Thread. After, Frame Processors run fully synchronous and always at the same FPS as the Camera. Two new functions have been introduced: * `runAtTargetFps(fps: number, func: () => void)`: Runs the given code as often as the given `fps`, effectively throttling it's calls. * `runAsync(frame: Frame, func: () => void)`: Runs the given function on a separate Thread for Frame Processing. A strong reference to the Frame is held as long as the function takes to execute. You can use `runAtTargetFps` to throttle calls to a specific API (e.g. if your Camera is running at 60 FPS, but you only want to run face detection at ~25 FPS, use `runAtTargetFps(25, ...)`.) You can use `runAsync` to run a heavy algorithm asynchronous, so that the Camera is not blocked while your algorithm runs. This is useful if your main sync processor draws something, and your async processor is doing some image analysis on the side. You can also combine both functions. Examples: ```js const frameProcessor = useFrameProcessor((frame) => { 'worklet' console.log("I'm running at 60 FPS!") }, []) ``` ```js const frameProcessor = useFrameProcessor((frame) => { 'worklet' console.log("I'm running at 60 FPS!") runAtTargetFps(10, () => { 'worklet' console.log("I'm running at 10 FPS!") }) }, []) ``` ```js const frameProcessor = useFrameProcessor((frame) => { 'worklet' console.log("I'm running at 60 FPS!") runAsync(frame, () => { 'worklet' console.log("I'm running on another Thread, I can block for longer!") }) }, []) ``` ```js const frameProcessor = useFrameProcessor((frame) => { 'worklet' console.log("I'm running at 60 FPS!") runAtTargetFps(10, () => { 'worklet' runAsync(frame, () => { 'worklet' console.log("I'm running on another Thread at 10 FPS, I can block for longer!") }) }) }, []) ```
* fix: Fix CI for "Build Android" * update versions * Update Gemfile.lock * format swift * fix: Fix swift lint * Update .swiftlint.yml * Use C++17 for lint * fix: Fix C++ lints
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as resolved.
This comment was marked as resolved.
* Create Shaders.ts * Add `previewType` and `enableFpsGraph` * Add RN Skia native dependency * Add Skia Preview View on iOS * Pass 1 * Update FrameHostObject.mm * Wrap Canvas * Lockfiles * fix: Fix stuff * chore: Upgrade RNWorklets * Add `previewType` to set the Preview * feat: Add Example * Update project.pbxproj * `enableFpsGraph` * Cache the `std::shared_ptr<FrameHostObject>` * Update CameraView+RecordVideo.swift * Update SkiaMetalCanvasProvider.mm * Android: Integrate Skia Dependency * fix: Use new Prefix * Add example for rendering shader * chore: Upgrade CameraX * Remove KTX * Enable `viewBinding` * Revert "Enable `viewBinding`" This reverts commit f2a603f. * Revert "chore: Upgrade CameraX" This reverts commit 8dc832c. * Remove unneeded `ProcessCameraProvider.getInstance()` call * fix: Add REA hotfix patch * fix: Fix FrameHostObject dead in runAsync * fix: Make `runAsync` run truly async by dropping new Frames while executing * chore: Upgrade RN Worklets to latest * chore: Upgrade RN Skia * Revert "Remove KTX" This reverts commit 253f586. * Make Skia optional in CMake * Fix import * Update CMakeLists.txt * Update build.gradle * Update CameraView.kt * Update CameraView.kt * Update CameraView.kt * Update Shaders.ts * Center Blur * chore: Upgrade RN Worklets * feat: Add `toByteArray()`, `orientation`, `isMirrored` and `timestamp` to `Frame` (#1487) * feat: Implement `orientation` and `isMirrored` on Frame * feat: Add `toArrayBuffer()` func * perf: Do faster buffer copy * feat: Implement `toArrayBuffer()` on Android * feat: Add `orientation` and `isMirrored` to Android * feat: Add `timestamp` to Frame * Update Frame.ts * Update JImageProxy.h * Update FrameHostObject.cpp * Update FrameHostObject.cpp * Update CameraPage.tsx * fix: Format Swift
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit
Ahh, thank you!
Edited the comment :)
Yep I mainly focused on iOS for now, am currently looking more into Android. thanks for the logs, will investigate! |
Update 21.2.2023:I just released VisionCamera v3.0.0-rc.2, which includes the following features:
Try it:
Things to test: all of the features above |
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
…me`'s lifecycle (#1488) * fix: fix C++ lint * fix: attach `InvalidateCacheOnDestroy` to `jsi::Runtime`
Okay I had a mistake in |
See #1376
Breaking Changes
runAsync
inside a Frame Processor__
prefix, but rather stored directly in theFrameProcessorPlugins
object exported by react-native-vision-camera. (e.g. replace__scanQRCodes(frame)
withFrameProcessorPlugins.scanQRCodes(frame)
)frameProcessorFps
no longer exists. UserunAtTargetFps
inside a Frame Processor to throttle some calls.onFrameProcessorPerformanceSuggestionAvailable
no longer exists. Use the FPS display (enableFpsGraph={true}
) to see how your Frame Processor performs over time. This is more in-line with how React Native works (Dev Tools / Perf Monitor)Progress
You can test the latest V3 release by creating a new RN project with RN 0.71 and installing VisionCamera + RNWorklets:
Things to test:
.tflite
model!! ✨ (see this PR for more info, will be a separate library soon)frame.toArrayBuffer()
to get the Frame's byte content in JSrunAtTargetFps
andrunAsync
in Frame ProcessorsOverall V3 is close to completion. I have a few things to do the coming days so not sure how much work I can put into this. If anyone wants to support the development of v3, I'd appreciate donations / sponsors: https://github.com/sponsors/mrousavy ❤️ :)
Related issues
features
preset
#117...and then pretty much every Android issue lol
.. maybe also (not tested):
SIGSEGV
on Android #1635[capture/inactive-source]
The recording failed because the source becomes inactive and stops sending frames. One case is that if camera is closed due to lifecycle stopped, the active recording will be finalized with this error, and the output will be generated, containing the frames produced before camera closing. Attempting to start a new recording will be finalized immediately if the source remains inactive and no output will be generated. #959