frameBufferToMat maybe crashes app

[oscarlundberg-sthlm]

Hi,

I'm using:
"react-native-fast-opencv": "^0.3.4"

Trying to develop for a physical iPhone using Expo. Sometimes the app works, but mostly it crashes on start up.
I've been looking like crazy, but manage to notice that commenting out the lines that uses OpenCV.frameBufferToMat() seemed to make the crash go away.

Upon searching a crash report from the iPhone, I found what's displayed in the attached image.
I'm more of a web developer, so I don't know much about this. But it seems there might be a memory issue?

[lukaszkurantdev]

Hi @oscarlundberg-sthlm, please provide code example and details about used additional libraries (Vision Camera).

[oscarlundberg-sthlm]

Hi! Absolutely.

Sorry for all the error-checks below, I've needed them, but they make the code a bit bloated to read.
If you disregard all of those, there really isn't that much code.

These are my related imports besides react-native-fast-opencv:

import {
  Camera,
  useCameraDevice,
  useCameraPermission,
  useFrameProcessor,
} from "react-native-vision-camera";
import { useResizePlugin } from "vision-camera-resize-plugin";

"react-native-vision-camera": "^4.6.3",
"vision-camera-resize-plugin": "^3.2.0"
"expo": "~52.0.23"
"react": "18.3.1"
"react-native": "0.76.5"

I call react-native-fast-opencv using a hook within a frameProcessor, the frameProcessor looks like this:

const frameProcessor = useFrameProcessor((frame) => {
    "worklet";

    const imageWidthRatio = frame.width / frame.height;
    const height = 20;
    const width = Math.round(height * imageWidthRatio);

    const resizedFrame = resize(frame, {
      scale: {
        width: width,
        height: height,
      },
      pixelFormat: "rgb",
      dataType: "uint8",
    });

    if (!resizedFrame) {
      console.error("Failed to resize frame");
      return;
    }

    if (!(resizedFrame instanceof Uint8Array)) {
      console.error("Resized frame is not a Uint8Array");
      return;
    }

    // Convert resizedFrame to plain array for compatibility with storing as previousFrame
    const resizedFrameArray = Array.from(resizedFrame);

    if (!previousFrame.current) {
      previousFrame.current = resizedFrameArray;
      return;
    }

    // Convert previousFrame back to Uint8Array for OpenCV usage
    const previousFrameAsUint8Array = new Uint8Array(
      Object.values(previousFrame.current)
    );

    if (!previousFrameAsUint8Array.length) {
      console.error("Previous frame is empty");
      return;
    }

    const motionDetected = useDetectMotion(
      previousFrameAsUint8Array,
      resizedFrame,
      width,
      height,
      30,
      0.05
    );

    if (motionDetected) {
      motionAction();
    }

    previousFrame.current = resizedFrameArray;
  }, []);

The hook using react-native-fast-opencv I call useDetectMotion, and it looks like this:

const useDetectMotion = (
  frame1: Uint8Array<ArrayBufferLike>,
  frame2: Uint8Array<ArrayBufferLike>,
  width: number,
  height: number,
  thresholdValue = 30,
  motionThreshold = 0.05
) => {
  "worklet";
  let motionDetected = false;

  // 1. Check if frames are valid
  if (!(frame1 instanceof Uint8Array)) {
    console.error("frame1 is not Uint8Array");
    return false;
  }
  if (!(frame2 instanceof Uint8Array)) {
    console.error("frame2 is not Uint8Array");
    return false;
  }

  try {
    // 2. Create Mats for frame1 and frame2
    const frame1Mat = OpenCV.frameBufferToMat(height, width, 3, frame1);
    const frame2Mat = OpenCV.frameBufferToMat(height, width, 3, frame2);

    if (!frame1Mat || !frame2Mat) {
      console.error("Failed to convert frames to Mat");
      OpenCV.clearBuffers();
      return false;
    }

    // 3. Compute absolute difference
    const diff = OpenCV.createObject(
      ObjectType.Mat,
      height,
      width,
      DataTypes.CV_8UC3
    );
    OpenCV.invoke("absdiff", frame1Mat, frame2Mat, diff);

    // 4. Convert difference to grayscale
    const grayDiff = OpenCV.createObject(
      ObjectType.Mat,
      height,
      width,
      DataTypes.CV_8UC1
    );
    OpenCV.invoke(
      "cvtColor",
      diff,
      grayDiff,
      ColorConversionCodes.COLOR_BGR2GRAY
    );

    // 5. Apply threshold
    const binaryDiff = OpenCV.createObject(
      ObjectType.Mat,
      height,
      width,
      DataTypes.CV_8UC1
    );
    const maxValue = 255;
    OpenCV.invoke(
      "threshold",
      grayDiff,
      binaryDiff,
      thresholdValue,
      maxValue,
      ThresholdTypes.THRESH_BINARY
    );

    // 6. Count non-zero pixels
    const nonZeroCount = OpenCV.invoke("countNonZero", binaryDiff);

    // 7. Calculate motion
    const totalPixels = height * width;
    const changeRatio = nonZeroCount.value / totalPixels;

    motionDetected = changeRatio > motionThreshold;
  } catch (error) {
    console.error("Error in useDetectMotion", error);
  }

  OpenCV.clearBuffers();
  return motionDetected;
};

Hope this clarifies things =)

[lukaszkurantdev]

@oscarlundberg-sthlm Sorry, for delay. I checked it using

const previousFrame = useSharedValue<number[] | null>(null);

from react-native-worklets-core and it works correctly for me.

Can you check it on more time on version 0.4.0 of FastOpenCV?

[oscarlundberg-sthlm]

Great! It does seem to work now.

Just out of curiosity, was there a memory issue you fixed in the source code, or just my use of useRef that messed things up?

[lukaszkurantdev]

@oscarlundberg-sthlm Great! It was fixed in #44 - there was a bug related to converting buffers to mat objects.