How to record video from a front camera without a mirror❓ #2338

HorbachAndrii · 2024-01-02T15:52:33Z

Question

I am analyzing recorded video and for me, it matters to record video without a mirror.

For IOS, I added some hack. I changed this connection.isVideoMirrored = false (from true to false) at node_modules/react-native-vision-camera/ios/Extensions/AVCaptureOutput+mirror.swift.

react-native-vision-camera/package/ios/Extensions/AVCaptureOutput+mirror.swift

Line 19 in ef4e9fa

connection.isVideoMirrored = true

But for Android, I don't know how to record video from a front camera without a mirror.

Maybe somebody knows how can I record video from a front camera without a mirror?

What I tried

I changing the Matrix for transformMatrix in VideoPipeline.kt and other stuff, but nothing helped

VisionCamera Version

3.6.16

Additional information

I am using Expo
I have read the Troubleshooting Guide
I agree to follow this project's Code of Conduct
I searched for similar questions in the issues page as well as in the discussions page and found none.

The text was updated successfully, but these errors were encountered:

HorbachAndrii · 2024-01-04T22:13:50Z

So I added also some hack that works for android.

VideoPipeline.kt node_modules/react-native-vision-camera/android/src/main/java/com/mrousavy/camera/core/VideoPipeline.kt

update this line

react-native-vision-camera/package/android/src/main/java/com/mrousavy/camera/core/VideoPipeline.kt

Line 165 in ef4e9fa

onFrame(transformMatrix)

onFrame(transformMatrix)

->

onFrame(transformMatrix, isMirrored)

update this line

react-native-vision-camera/package/android/src/main/java/com/mrousavy/camera/core/VideoPipeline.kt

Line 212 in ef4e9fa

private external fun onFrame(transformMatrix: FloatArray)

private external fun onFrame(transformMatrix: FloatArray)

->

private external fun onFrame(transformMatrix: FloatArray, isMirrored: Boolean)

VideoPipeline.cpp node_modules/react-native-vision-camera/android/src/main/cpp/VideoPipeline.cpp

update this line

react-native-vision-camera/package/android/src/main/cpp/VideoPipeline.cpp

Line 73 in ef4e9fa

void VideoPipeline::onFrame(jni::alias_ref<jni::JArrayFloat> transformMatrixParam) {

void VideoPipeline::onFrame(jni::alias_ref<jni::JArrayFloat> transformMatrixParam) {

->

void VideoPipeline::onFrame(jni::alias_ref<jni::JArrayFloat> transformMatrixParam, bool isMirrored) {

update this line

react-native-vision-camera/package/android/src/main/cpp/VideoPipeline.cpp

Line 82 in ef4e9fa

_recordingSessionOutput->renderTextureToSurface(texture, transformMatrix);

_recordingSessionOutput->renderTextureToSurface(texture, transformMatrix);

->

_recordingSessionOutput->renderTextureToSurface(texture, transformMatrix, isMirrored);

VideoPipeline.h node_modules/react-native-vision-camera/android/src/main/cpp/VideoPipeline.h

update this line

react-native-vision-camera/package/android/src/main/cpp/VideoPipeline.h

Line 39 in ef4e9fa

void onFrame(jni::alias_ref<jni::JArrayFloat> transformMatrix);

void onFrame(jni::alias_ref<jni::JArrayFloat> transformMatrix);

->

void onFrame(jni::alias_ref<jni::JArrayFloat> transformMatrix, bool isMirrored);

OpenGLRenderer.cpp node_modules/react-native-vision-camera/android/src/main/cpp/OpenGLRenderer.cpp

update this line

react-native-vision-camera/package/android/src/main/cpp/OpenGLRenderer.cpp

Line 46 in ef4e9fa

void OpenGLRenderer::renderTextureToSurface(const OpenGLTexture& texture, float* transformMatrix) {

void OpenGLRenderer::renderTextureToSurface(const OpenGLTexture& texture, float* transformMatrix) {

->

void OpenGLRenderer::renderTextureToSurface(const OpenGLTexture& texture, float* transformMatrix, bool isMirrored) {

update this line

react-native-vision-camera/package/android/src/main/cpp/OpenGLRenderer.cpp

Line 68 in ef4e9fa

_passThroughShader.draw(texture, transformMatrix);

_passThroughShader.draw(texture, transformMatrix);

->

_passThroughShader.draw(texture, transformMatrix, isMirrored);

OpenGLRenderer.h node_modules/react-native-vision-camera/android/src/main/cpp/OpenGLRenderer.h

update this line

react-native-vision-camera/package/android/src/main/cpp/OpenGLRenderer.h

Line 36 in ef4e9fa

void renderTextureToSurface(const OpenGLTexture& texture, float* transformMatrix);

void renderTextureToSurface(const OpenGLTexture& texture, float* transformMatrix);

->

void renderTextureToSurface(const OpenGLTexture& texture, float* transformMatrix, bool isMirrored);

PassThroughShader.cpp node_modules/react-native-vision-camera/android/src/main/cpp/PassThroughShader.cpp

update this line

react-native-vision-camera/package/android/src/main/cpp/PassThroughShader.cpp

Line 27 in ef4e9fa

void PassThroughShader::draw(const OpenGLTexture& texture, float* transformMatrix) {

 void PassThroughShader::draw(const OpenGLTexture& texture, float* transformMatrix) {

->

void PassThroughShader::draw(const OpenGLTexture& texture, float* transformMatrix, bool isMirrored) {

update this line

react-native-vision-camera/package/android/src/main/cpp/PassThroughShader.cpp

Line 30 in ef4e9fa

_programId = createProgram();

_programId = createProgram();

->

_programId = createProgram(isMirrored);

update this line

react-native-vision-camera/package/android/src/main/cpp/PassThroughShader.cpp

Line 87 in ef4e9fa

GLuint PassThroughShader::createProgram() {

GLuint PassThroughShader::createProgram() {

->

GLuint PassThroughShader::createProgram(bool isMirrored) {

update this line

react-native-vision-camera/package/android/src/main/cpp/PassThroughShader.cpp

Line 89 in ef4e9fa

GLuint fragmentShader = loadShader(GL_FRAGMENT_SHADER, FRAGMENT_SHADER);

 GLuint fragmentShader = loadShader(GL_FRAGMENT_SHADER, FRAGMENT_SHADER);

->

GLuint fragmentShader;
if (isMirrored) {
  fragmentShader = loadShader(GL_FRAGMENT_SHADER, FRAGMENT_SHADER_FLIPPED);
} else {
  fragmentShader = loadShader(GL_FRAGMENT_SHADER, FRAGMENT_SHADER);
}

PassThroughShader.h node_modules/react-native-vision-camera/android/src/main/cpp/PassThroughShader.h

update this line after

react-native-vision-camera/package/android/src/main/cpp/PassThroughShader.h

Line 32 in ef4e9fa

void draw(const OpenGLTexture& texture, float* transformMatrix);

void draw(const OpenGLTexture& texture, float* transformMatrix);

->

void draw(const OpenGLTexture& texture, float* transformMatrix, bool isisMirrored);

update this line after

react-native-vision-camera/package/android/src/main/cpp/PassThroughShader.h

Line 37 in ef4e9fa

static GLuint createProgram();

static GLuint createProgram();

->

static GLuint createProgram(bool isisMirrored);

add code after this FRAGMENT_SHADER

react-native-vision-camera/package/android/src/main/cpp/PassThroughShader.h

Lines 72 to 81 in ef4e9fa

    
             static constexpr char FRAGMENT_SHADER[] = R"( 
        
               #extension GL_OES_EGL_image_external : require 
        
               precision mediump float; 
        
               varying vec2 vTexCoord; 
        
               uniform samplerExternalOES uTexture; 
        
               void main() { 
        
                   gl_FragColor = texture2D(uTexture, vTexCoord); 
        
               } 
        
             )";

static constexpr char FRAGMENT_SHADER_FLIPPED[] = R"(
    #extension GL_OES_EGL_image_external : require
    precision mediump float;
    varying vec2 vTexCoord;
    uniform samplerExternalOES uTexture;

    void main() {
        gl_FragColor = texture2D(uTexture, vec2(vTexCoord.x, 1.0 - vTexCoord.y));
    }
  )";

CyxouD · 2024-04-30T09:16:01Z

It doesn't work in my case on iOS with the same react-native-vision-camera. Could you say what iOS device you use? Also, could you send your Camera component JSX (if something differs here)?

HorbachAndrii · 2024-04-30T13:02:01Z

It doesn't work in my case on iOS with the same react-native-vision-camera. Could you say what iOS device you use? Also, could you send your Camera component JSX (if something differs here)?

What VisionCamera Version are you using?

Could you say what iOS device you use?

it works for all IOS devices

CyxouD · 2024-04-30T14:48:09Z

@HorbachAndrii thank you for your quick response!

What VisionCamera Version are you using?

I tried with 3.6.16 (as you mentioned) and 3.9.2. I tested with iPhone 11.

HorbachAndrii · 2024-04-30T15:44:52Z

here is my AVCaptureOutput+mirror.swift file

//
//  AVCaptureOutput+mirror.swift
//  mrousavy
//
//  Created by Marc Rousavy on 18.01.21.
//  Copyright © 2021 mrousavy. All rights reserved.
//

import AVFoundation

extension AVCaptureOutput {
  /**
   Mirrors the video output if possible.
   */
  func mirror() {
    connections.forEach { connection in
      if connection.isVideoMirroringSupported {
        connection.automaticallyAdjustsVideoMirroring = false
        connection.isVideoMirrored = false
      }
    }
  }

  /**
   Sets the target orientation of the video output.
   This does not always physically rotate image buffers.

   - For Preview, an orientation hint is used to rotate the layer/view itself.
   - For Photos, an EXIF tag is used.
   - For Videos, the buffers are physically rotated if available, since we use an AVCaptureVideoDataOutput instead of an AVCaptureMovieFileOutput.
   */
  func setOrientation(_ orientation: Orientation) {
    // Set orientation for each connection
    connections.forEach { connection in
      #if swift(>=5.9)
        if #available(iOS 17.0, *) {
          // Camera Sensors are always in landscape rotation (90deg).
          // We are setting the target rotation here, so we need to rotate by landscape once.
          let cameraOrientation = orientation.rotateBy(orientation: .landscapeLeft)
          let degrees = cameraOrientation.toDegrees()

          // TODO: Don't rotate the video output because it adds overhead. Instead just use EXIF flags for the .mp4 file if recording.
          //       Does that work when we flip the camera?
          if connection.isVideoRotationAngleSupported(degrees) {
            connection.videoRotationAngle = degrees
          }
        } else {
          if connection.isVideoOrientationSupported {
            connection.videoOrientation = orientation.toAVCaptureVideoOrientation()
          }
        }
      #else
        if connection.isVideoOrientationSupported {
          connection.videoOrientation = orientation.toAVCaptureVideoOrientation()
        }
      #endif
    }
  }
}

HorbachAndrii · 2024-04-30T15:59:48Z

@HorbachAndrii thank you for your quick response!

What VisionCamera Version are you using?

I tried with 3.6.16 (as you mentioned) and 3.9.2. I tested with iPhone 11.

Did you 'Clean Build Folder' before building the app?

CyxouD · 2024-05-01T13:26:43Z

Did you 'Clean Build Folder' before building the app?
@HorbachAndrii Thank you! I tried it with 3.9.1 and 3.9.2, but unfortunately it didn't work. I didn't clean the build folder with 3.6.16 because it had another bugs, but I tested with logs that changes were reflected.
I will check if it works with other versions of iPhone (not 11)

mrousavy · 2024-07-02T10:02:16Z

Hey all! I just spent some time to build a new feature: I created a PR to add a isMirrored prop to Camera: #3036

With this prop you can enable or disable output mirroring 🎉🥳

If you appreciate my time and dedication towards building new features and constantly improving this library, please consider sponsoring me on GitHub :)

CyxouD · 2024-07-03T08:28:51Z

@mrousavy thank you for introducing new features! However, in this issue, we needed Preview to be not mirrored in the front camera, but it isn't supported:

To avoid dizzyness for users, the Preview will always be mirrored in the front Camera, but not in the back Camera. This is the same behaviour as in the native iOS Camera app.

mrousavy · 2024-07-03T08:34:44Z

This is intentional - if the preview was mirrored the app would be pretty unusable because every movement goes in the wrong direction. All camera apps mirror the front preview.

If you really really need the preview not mirrored for whatever reason, you can just apply a scaleX: -1 transform to the <Camera> view style.

HorbachAndrii added the 💭 question Further information is requested label Jan 2, 2024

HorbachAndrii closed this as completed Jan 4, 2024

mrousavy mentioned this issue Jul 2, 2024

feat: Add isMirrored prop #3036

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How to record video from a front camera without a mirror❓ #2338

How to record video from a front camera without a mirror❓ #2338

HorbachAndrii commented Jan 2, 2024 •

edited

Loading

HorbachAndrii commented Jan 4, 2024 •

edited

Loading

CyxouD commented Apr 30, 2024

HorbachAndrii commented Apr 30, 2024 •

edited

Loading

CyxouD commented Apr 30, 2024 •

edited

Loading

HorbachAndrii commented Apr 30, 2024

HorbachAndrii commented Apr 30, 2024

CyxouD commented May 1, 2024

mrousavy commented Jul 2, 2024

CyxouD commented Jul 3, 2024

mrousavy commented Jul 3, 2024

How to record video from a front camera without a mirror❓ #2338

How to record video from a front camera without a mirror❓ #2338

Comments

HorbachAndrii commented Jan 2, 2024 • edited Loading

Question

What I tried

VisionCamera Version

Additional information

HorbachAndrii commented Jan 4, 2024 • edited Loading

CyxouD commented Apr 30, 2024

HorbachAndrii commented Apr 30, 2024 • edited Loading

CyxouD commented Apr 30, 2024 • edited Loading

HorbachAndrii commented Apr 30, 2024

HorbachAndrii commented Apr 30, 2024

CyxouD commented May 1, 2024

mrousavy commented Jul 2, 2024

CyxouD commented Jul 3, 2024

mrousavy commented Jul 3, 2024

HorbachAndrii commented Jan 2, 2024 •

edited

Loading

HorbachAndrii commented Jan 4, 2024 •

edited

Loading

HorbachAndrii commented Apr 30, 2024 •

edited

Loading

CyxouD commented Apr 30, 2024 •

edited

Loading