[Bug] Render image on Iphone SE w IOS 17.

[gosvig123]

Describe the Bug

The issue is that the image seem to load, but the canvas or image loaded is not vissible. This is only a problem when trying to load an image from my iphone SE, have tried different OS variations from 17.1 and up.
When trying to run the same code locally in the browser, or emulating the Iphone SE from the chrome dev tools the issue does not occur.

On the phone the loading indicate that the image has loaded, and the tool still works in the sense that windowing can be activated, and change the settings within the canvas that has no content.

Steps to Reproduce

Try to load an image from an Iphone SE with OS 17.1 either using Ngrok or forward the tunnel to the device from the laptop running the.

The current behavior

Everything works well, except when tested locally on an Iphone SE (have not tested other actual devices)

The expected behavior

For the Image to display inside the canvas after being loaded.

OS

IOS 17.1

Node version

18.19

Browser

Chrome or Safari (tested both)

[abustany]

This also happens with https://www.cornerstonejs.org/live-examples/stackbasic , which I suppose is using the latest cornerstone version (I'm on iOS 17.3.1). I have the same issue with cornerstone 1.58.4.

[abustany]

https://www.cornerstonejs.org/live-examples/webloader works though, so maybe this is an issue with the dicom loader?

[abustany]

From what I could gather so far:

• the DICOM loader seems to do its job fine and is not at fault
• forcing CPU rendering works around the issue (but is not a great solution)
• https://kitware.github.io/vtk-js/examples/MultiSliceImageMapper.html works on iOS - so not a core VTK issue?
• updating vtk.js to latest in cornerstone does not fix the issue

so something in the GPU rendering pipeline is broken on iOS, going down the rabbit hole...

When rendering a web image, we have an Uint8Array on 3 components (R, G, B). When rendering a CT file, we have a Float32Array on 1 component. I tried remapping Float32Array 1-comp pixel data to Uint8Array 1-comp inside updateVTKImageDataWithCornerstoneImage to see if it makes a difference, but no success so far...

[abustany]

I hacked updateVTKImageDataWithCornerstoneImage to remap single channel (Float32Array numComponents=1) to three-channel images (Uint8Array numComponents=3), and the image displays on iOS correctly now. That doesn't really tell what the problem is, much less what the fix is, but at least I know what triggers the issue.

Update: I also tried to remap to 3xf32, and that again does not render correctly...

[abustany]

I now have a reduced test case using only vtk.js:

import '@kitware/vtk.js/Rendering/Profiles/Volume';
import vtkImageData from '@kitware/vtk.js/Common/DataModel/ImageData';
import vtkDataArray from '@kitware/vtk.js/Common/Core/DataArray';
import vtkFullScreenRenderWindow from '@kitware/vtk.js/Rendering/Misc/FullScreenRenderWindow';
import vtkImageMapper from '@kitware/vtk.js/Rendering/Core/ImageMapper';
import vtkImageSlice from '@kitware/vtk.js/Rendering/Core/ImageSlice';
import {SlicingMode} from '@kitware/vtk.js/Rendering/Core/ImageMapper/Constants';

function main() {
  const imageSize = 512;
  const pixels = new Float32Array(imageSize*imageSize);

  for (let y = 0; y < imageSize; ++y) {
    for (let x = 0; x < imageSize; ++x) {
      pixels[y*imageSize+x] = (x*y)%256;
    }
  }

  const scalarArray = vtkDataArray.newInstance({
    name: 'Pixels',
    numberOfComponents: 1,
    values: pixels,
  });

  const imageData = vtkImageData.newInstance();

  imageData.setDimensions([512, 512, 1]);
  imageData.setSpacing([1, 1, 1]);
  imageData.setOrigin([imageSize>>1, imageSize>>1, 1]);
  imageData.getPointData().setScalars(scalarArray);

  const mapper = vtkImageMapper.newInstance();
  mapper.setInputData(imageData);
  mapper.setSliceAtFocalPoint(true);
  mapper.setSlicingMode(SlicingMode.Z);

  const actor = vtkImageSlice.newInstance();
  actor.getProperty().setColorWindow(255);
  actor.getProperty().setColorLevel(127);
  actor.setMapper(mapper);

  const fullScreenRenderer = vtkFullScreenRenderWindow.newInstance({
    container: document.getElementById("imageContainer")!,
  });

  const renderer = fullScreenRenderer.getRenderer();
  renderer.addActor(actor);
  renderer.resetCamera();

  const renderWindow = fullScreenRenderer.getRenderWindow();
  renderWindow.getInteractor().setDesiredUpdateRate(30);
  renderWindow.render();
}

main();

This code display correctly on desktop, but shows a black square on iOS.

[abustany]

I added a hacky patch to the vtk bug thread, which fixes loading regular DICOM images for me on iOS. It's not meant as a bullet proof or general solution, but I'm posting it here in case it helps someone else :)

[gosvig123]

@sedghi , any news on this one? Thanks!

[abustany]

@gosvig123 another easy workaround is to use the preferSizeOverAccuracy Cornerstone option, which will make it decode DICOM pixel data to Int16 instead. With that, you probably don't even need the VTK patch.

[gosvig123]

@abustany , that looked super interesting and just got excited so thank you very much for the pointer!

Just did some quick testing and can only get it to function with 2d images though, i know you mentioned you are not using volume, but did you by any chance have it working with volume or see any issue with it (really appreciate all the help and do have a working patch, but would be nice to not have to patch.)

[sedghi]

Hey guys, sorry we are in the middle of a release, and didn't have time to look into this
Let me read and reply

[sedghi]

Based on my investigation the issue is Float Linear interpolation (OES_texture_float_linear) is not supported on iOS devices and was incorrectly reported as supported up until recently in iOS 17

https://bugs.webkit.org/show_bug.cgi?id=264404#c2

[sedghi]

I created this PR which would be good enough for a lot of use cases (except the PT Scaled data)

#1212

[abustany]

Nice, thanks for this ! I’ll try to give it a try in the coming days