Slow Processing Time

[nicolazreinh]

Description

I am using the Angular version of background-removal-js for background removal in an image capture application. The library works as expected but the processing time to remove the background is quite slow. This impacts the user experience significantly.

Steps to Reproduce:

1. Capture an image using a canvas in an Angular application.
2. Call the removeBackground function with the image blob.
3. Observe the time it takes to process the background removal.

Expected Behavior:

The background removal should be faster, or at least there should be an option to use a more optimized/smaller model to reduce the processing time.

Actual Behavior:

• The processing time takes significantly longer than expected.
• Even when using the isnet_quint8 model, which is supposed to be optimized, the performance issue persists.

Logs:

The following logs show time metrics related to the model download and processing. These details might help in diagnosing the issue:

Time: 1338
Downloading fetch:/models/isnet_quint8: 4194304 of 44348940
Time: 68
Downloading fetch:/models/isnet_quint8: 8388608 of 44348940
Time: 160
Downloading fetch:/models/isnet_quint8: 12582912 of 44348940
...
Downloading compute:decode: 0 of 4
Time: 7
Downloading compute:inference: 1 of 4
Time: 22251
Downloading compute:mask: 2 of 4
Time: 1
Downloading compute:encode: 4 of 4

Total processing times:

• Download of the model: ~3.5 seconds
• Inference (main processing step): ~22 seconds

My Setup:

• Angular version: 17
• background-removal-js version: latest
• Browser: Chrome
• Platform: Windows

Code Snippet:

applyFilter(): void {
    if (this.canvas === null) {
      return;
    }

    let now = new Date().getTime();

    this.canvas.toBlob((blob: Blob | null) => {
      if (blob === null) {
        console.error(`Blob canvas is null`);
        return;
      }
      removeBackground(blob, {
        model: 'isnet_quint8', // Using the lighter model
        progress: (key, current, total) => {
          const end = new Date().getTime();
          console.log(`Time: ${end - now}`)
          now = end;
          console.log(`Processing ${key}: ${current} of ${total}`);
        }
      }).then((blob: Blob) => {
        const reader = new FileReader();
        reader.readAsDataURL(blob);
        reader.onloadend = function () {
          const img = new Image();
          img.onload = () => {
            const context = this.canvas.getContext('2d');
            context?.clearRect(0, 0, this.canvas.width, this.canvas.height);
            context?.drawImage(img, 0, 0, img.width, img.height);
          }
          img.src = reader.result as string;
        }
      }).catch((error: any) => {
        console.error(`Failed to remove background: ${error.message}`);
      });
    });
}

Suggestions:

• Optimize the current model for faster performance.
• Provide smaller, more optimized models to choose from.
• Improve documentation on expected performance based on model size and image resolution.

[a-hariti]

did you try with WebGPU @nicolazreinh?

const config: Config = {
  device: "gpu"
}