[WIP] ios Image segmenter sample #292

st-tuanmai · 2023-12-12T07:29:51Z

Description

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context.

Fixes # (issue)

Checklist

Please ensure the following items are complete before submitting a pull request:

My code follows the code style of the project.
I have updated the documentation (if applicable).
I have added tests to cover my changes.

Type of Change

Please check the relevant option below:

Bug fix (non-breaking change which fixes an issue)
Documentation update (non-breaking change which updates documentation)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)

Screenshots

If applicable, please add screenshots to help explain your changes.

Additional Notes

Add any additional information or context about the pull request here.

priankakariat

@st-tuanmai Can you implement the landscape orientations ?

priankakariat

@st-tuanmai The image selected from the gallery is not displayed on the screen. The screen remains blank

st-tuanmai · 2024-01-10T04:49:29Z

@st-tuanmai The image selected from the gallery is not displayed on the screen. The screen remains blank

Can you help me check the result of the image segmented api when segmenting UIImage when the orientation is left or right, the result returned the wrong width, height, and orientation is reverse (up -> down).
Thank you.

priankakariat · 2024-01-12T13:10:32Z

@st-tuanmai The image selected from the gallery is not displayed on the screen. The screen remains blank

Can you help me check the result of the image segmented api when segmenting UIImage when the orientation is left or right, the result returned the wrong width, height, and orientation is reverse (up -> down). Thank you.

I can help you. But could you check why an image is not displayed on screen when it is selected ?

st-tuanmai · 2024-01-16T01:46:35Z

I can help you. But could you check why an image is not displayed on screen when it is selected ?

@priankakariatyml I will check again, it works on my device. could you use the latest code? i think some codes is missing.
thanks you!

st-tuanmai · 2024-01-16T03:16:00Z

I can help you. But could you check why an image is not displayed on screen when it is selected ?

@priankakariatyml
I have reproduced the issues when segmenting the image many times, probably related to freeing memory when rendering the image, I will try to fix it.

st-tuanmai · 2024-01-16T09:14:17Z

@priankakariatyml Please check the latest code, I fixed the image orientation and memory issues.
Next, I will fix the code conventions and change UI before sending you the final version.
Thanks you!

priankakariat · 2024-01-18T17:45:08Z

@priankakariatyml Please check the latest code, I fixed the image orientation and memory issues.
Next, I will fix the code conventions and change UI before sending you the final version.
Thanks you!

Hi I can see the results of segmentation of gallery images now. But the UI seems to be out of place. I can’t see the bottom bar button. My device is iPhone 11 Pro Max. There is only a white band at the bottom.

Also after performing inference on images from gallery the resulting image seems to be a little stretched. Can you check if the uiimageview is using the correct content mode.

Also I don’t think the orientation fix has been pushed. The app is still restricted to portrait I think.

st-tuanmai · 2024-01-19T02:15:57Z

Hi I can see the results of segmentation of gallery images now. But the UI seems to be out of place. I can’t see the bottom bar button. My device is iPhone 11 Pro Max. There is only a white band at the bottom. Also after performing inference on images from gallery the resulting image seems to be a little stretched. Can you check if the uiimageview is using the correct content mode.

Thanks @priankakariatyml
I will check the UI.

Also I don’t think the orientation fix has been pushed. The app is still restricted to portrait I think.

I have fixed the image to portrait and passed in the fixed orientation as .up.

priankakariat

@st-tuanmai Can you replicate the bottom bar exactly like android. Right now the iOS UI only shows inference time. Android app has controls to switch between the model and change the delegate. These bottom bar is collapsible like in the image classification sample. Can you refer to the Android app and implement these changes ?

priankakariat · 2024-01-24T18:03:58Z

Hi I can see the results of segmentation of gallery images now. But the UI seems to be out of place. I can’t see the bottom bar button. My device is iPhone 11 Pro Max. There is only a white band at the bottom. Also after performing inference on images from gallery the resulting image seems to be a little stretched. Can you check if the uiimageview is using the correct content mode.

Thanks @priankakariatyml I will check the UI.

Also I don’t think the orientation fix has been pushed. The app is still restricted to portrait I think.

I have fixed the image to portrait and passed in the fixed orientation as .up.

Is there a reason why other orientations are not supported ? Like the previous samples

st-tuanmai · 2024-01-25T02:49:00Z

Is there a reason why other orientations are not supported ? Like the previous samples

I'm having issues with the live camera when rendering in landscape orientation. I will change to support it soon.

priankakariat · 2024-02-07T13:37:48Z

examples/image_segmentation/ios/ImageSegmenter/Configurations/DefaultConstants.swift

+// MARK: Define default constants
+struct DefaultConstants {
+
+  static let lineWidth: CGFloat = 2


Except for the model path, remove the other constants since they seem to be unused

priankakariat · 2024-02-07T13:50:49Z

examples/image_segmentation/ios/ImageSegmenter/ViewContoller/MediaLibraryViewController.swift

+        let sourceImage = request.sourceImage
+        let time = Float((request.compositionTime - timeRange.start).seconds)
+        let cgimage = self.render.getCGImmage(ciImage: sourceImage)
+        guard let resultBundle = self.imageSegmenterService?.segment(by: cgimage, orientation: .up, timeStamps: Int(time * 1000)) else {


Like in the previous samples, please encapsulate all this code in the ImageSegmenterSevice method segment(
videoAsset: AVAsset,
durationInMilliseconds: Double,
inferenceIntervalInMilliseconds: Double) async -> ResultBundle?

examples/image_segmentation/ios/ImageSegmenter/ViewContoller/MediaLibraryViewController.swift

priankakariat · 2024-02-07T14:00:33Z

@priankakariatyml Please check the latest code, I fixed the image orientation and memory issues.
Next, I will fix the code conventions and change UI before sending you the final version.
Thanks you!

Hi I can see the results of segmentation of gallery images now. But the UI seems to be out of place. I can’t see the bottom bar button. My device is iPhone 11 Pro Max. There is only a white band at the bottom. Also after performing inference on images from gallery the resulting image seems to be a little stretched. Can you check if the uiimageview is using the correct content mode.

Also I don’t think the orientation fix has been pushed. The app is still restricted to portrait I think.

@st-tuanmai Could you resolve the issue with the bottom bar and adding the capability to choose the model?

priankakariat · 2024-02-07T14:01:08Z

Is there a reason why other orientations are not supported ? Like the previous samples

I'm having issues with the live camera when rendering in landscape orientation. I will change to support it soon.

We can support it after the app is made feature complete

priankakariat · 2024-02-07T14:02:34Z

examples/image_segmentation/ios/ImageSegmenter/Services/Render.swift

@@ -0,0 +1,578 @@
+


Rename file to SegmentedImageRenderer

priankakariat · 2024-02-07T14:02:49Z

examples/image_segmentation/ios/ImageSegmenter/Services/Render.swift

+import MetalPerformanceShaders
+import MetalKit
+
+class Render {


s/Render/SegmentedImageRenderer

priankakariat · 2024-02-07T14:03:19Z

examples/image_segmentation/ios/ImageSegmenter/Units/MergeColorShaders.metal

+#include <metal_stdlib>
+using namespace metal;
+
+half4 choseColor(half4 color1, half4 color2, float cf) {


s/choseColor/ChooseColor

priankakariat · 2024-02-07T14:09:46Z

examples/image_segmentation/ios/ImageSegmenter/Services/Render.swift

+    isPrepared = false
+  }
+
+  func render(image: UIImage, segmentDatas: UnsafePointer<Float32>?) -> UIImage? {


s/segmentDatas/confidenceMasks

priankakariat · 2024-02-07T14:15:45Z

examples/image_segmentation/ios/ImageSegmenter/Services/Render.swift

+    isPrepared = false
+  }
+
+  func render(image: UIImage, segmentDatas: UnsafePointer<Float32>?) -> UIImage? {


Why is only the mask at index 0 taken for display ?. This will work for selfie segmentation. But if you use a model like deeplabv3 which will provide segmentation masks for all the categories, this won't work.
The sample needs to support Deeply V3 too. The user should be allowed to switch between deeplabv3 and selfie segmenter model in the bottom bar.
Could you please implement this ?

I will do it.

priankakariat · 2024-02-07T14:17:30Z

examples/image_segmentation/ios/ImageSegmenter/Services/ImageSegmenterService.swift

+    }
+
+  func segment(
+    by videoFrame: CGImage,


priankakariat · 2024-02-07T14:22:23Z

examples/image_segmentation/ios/ImageSegmenter/ViewContoller/MediaLibraryViewController.swift

+    let naturalSize = videoTrack.naturalSize
+    let size = videoTrack.naturalSize.applying(videoTrack.preferredTransform)
+    let newSize = CGSize(width: abs(size.width), height: abs(size.height))
+    var needChangeWidthHeight = false


Remove this variable and directly return using the condition
return (formatDescription, naturalSize.width == newSize.height)

priankakariat · 2024-02-07T14:23:48Z

examples/image_segmentation/ios/ImageSegmenter/ViewContoller/MediaLibraryViewController.swift

+      assetReader.add(trackOutput)
+
+      // Start the asset reader
+      if assetReader.startReading() {


Convert the nested ifs in this block to guard let statements to improve readability

priankakariat · 2024-02-07T14:24:35Z

examples/image_segmentation/ios/ImageSegmenter/ViewContoller/MediaLibraryViewController.swift

+  }
+
+  private func playVideoAnDetect(asset: AVAsset) {
+    let videoDescription = getVideoFormatDescription(from: asset)


s/playVideoAnDetect/playVideoAndSegment

priankakariat · 2024-02-07T14:25:11Z

examples/image_segmentation/ios/ImageSegmenter/ViewContoller/MediaLibraryViewController.swift

+            self.hideProgressView()
+            self.render.prepare(with: image.size, outputRetainedBufferCountHint: 3)
+            self.inferenceResultDeliveryDelegate?.didPerformInference(result: resultBundle)
+            let marks = imageSegmenterResult.confidenceMasks


s/marks/masks

priankakariat · 2024-02-07T14:26:03Z

examples/image_segmentation/ios/ImageSegmenter/ViewContoller/MediaLibraryViewController.swift

+            self.render.prepare(with: image.size, outputRetainedBufferCountHint: 3)
+            self.inferenceResultDeliveryDelegate?.didPerformInference(result: resultBundle)
+            let marks = imageSegmenterResult.confidenceMasks
+            let _mark = marks![0]


Please see my comment here https://github.com/googlesamples/mediapipe/pull/292/files#r1481541212

st-tuanmai · 2024-02-23T07:10:43Z

@priankakariatyml
Hi Prianka,
Out of memory error appears when I segment using deeplabv3 model with large image ((3024.0, 4032.0)).
Can you help me check it?

PaulTR · 2024-03-18T14:03:25Z

@priankakariatyml Hey Prianka, any updates here?

priankakariat · 2024-03-18T14:12:01Z

@priankakariatyml Hey Prianka, any updates here?
I'll review this again. Was pretty busy with the Llm APIs. Haven't come back to this.

PaulTR · 2024-03-25T19:19:15Z

Sorry, merged in the other PRs that were going in. Can you please merge main back into this PR? Thanks

examples/image_segmentation/ios/RunScripts/download_models.sh

# Conflicts: # examples/object_detection/ios/ObjectDetector/ViewControllers/BottomSheetViewController.swift

st-tuanmai added 3 commits December 6, 2023 16:29

demo image segmenter

bbcbaab

add render video

0c10038

video segment

0525a91

priankakariat reviewed Jan 4, 2024

View reviewed changes

test render image

f57e8de

st-tuanmai added 2 commits January 16, 2024 15:59

fix image show black

ede87e8

rotate image

31b79fb

change name, send image width to shader

92a9252

fix UI

19a7ea5

priankakariat reviewed Jan 24, 2024

View reviewed changes

st-tuanmai added 2 commits January 31, 2024 14:31

suport lanscape orientation

4074583

remove unnecessary code

bcc9a39

priankakariat reviewed Feb 7, 2024

View reviewed changes

examples/image_segmentation/ios/ImageSegmenter/ViewContoller/MediaLibraryViewController.swift Outdated Show resolved Hide resolved

priankakariat reviewed Feb 7, 2024

View reviewed changes

test using deeplabv3

7716c90

update to 0.10.12

6e3cf7a

st-tuanmai added 2 commits March 20, 2024 14:53

add change model function

09c7e18

add chose delegate

24103fd

PaulTR reviewed Mar 25, 2024

View reviewed changes

examples/image_segmentation/ios/RunScripts/download_models.sh Show resolved Hide resolved

st-tuanmai added 2 commits March 26, 2024 09:54

Merge branch 'mediapipe_main' into feature/image_segmenter

6888201

# Conflicts: # examples/object_detection/ios/ObjectDetector/ViewControllers/BottomSheetViewController.swift

Add download deeplab v3 model ro RunScripts file

34d5176

PaulTR merged commit 0825e07 into google-ai-edge:main Mar 26, 2024
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] ios Image segmenter sample #292

[WIP] ios Image segmenter sample #292

st-tuanmai commented Dec 12, 2023

priankakariat left a comment •

edited

Loading

priankakariat left a comment

st-tuanmai commented Jan 10, 2024 •

edited

Loading

priankakariat commented Jan 12, 2024

st-tuanmai commented Jan 16, 2024 •

edited

Loading

st-tuanmai commented Jan 16, 2024 •

edited

Loading

st-tuanmai commented Jan 16, 2024

priankakariat commented Jan 18, 2024 •

edited

Loading

st-tuanmai commented Jan 19, 2024 •

edited

Loading

priankakariat left a comment

priankakariat commented Jan 24, 2024

st-tuanmai commented Jan 25, 2024

priankakariat Feb 7, 2024

priankakariat Feb 7, 2024

priankakariat commented Feb 7, 2024 •

edited

Loading

priankakariat commented Feb 7, 2024

priankakariat Feb 7, 2024

priankakariat Feb 7, 2024

priankakariat Feb 7, 2024

priankakariat Feb 7, 2024

priankakariat Feb 7, 2024 •

edited

Loading

st-tuanmai Feb 15, 2024

priankakariat Feb 7, 2024

priankakariat Feb 7, 2024

priankakariat Feb 7, 2024

priankakariat Feb 7, 2024

priankakariat Feb 7, 2024

priankakariat Feb 7, 2024

st-tuanmai commented Feb 23, 2024

PaulTR commented Mar 18, 2024

priankakariat commented Mar 18, 2024

PaulTR commented Mar 25, 2024

[WIP] ios Image segmenter sample #292

[WIP] ios Image segmenter sample #292

Conversation

st-tuanmai commented Dec 12, 2023

Description

Checklist

Type of Change

Screenshots

Additional Notes

priankakariat left a comment • edited Loading

Choose a reason for hiding this comment

priankakariat left a comment

Choose a reason for hiding this comment

st-tuanmai commented Jan 10, 2024 • edited Loading

priankakariat commented Jan 12, 2024

st-tuanmai commented Jan 16, 2024 • edited Loading

st-tuanmai commented Jan 16, 2024 • edited Loading

st-tuanmai commented Jan 16, 2024

priankakariat commented Jan 18, 2024 • edited Loading

st-tuanmai commented Jan 19, 2024 • edited Loading

priankakariat left a comment

Choose a reason for hiding this comment

priankakariat commented Jan 24, 2024

st-tuanmai commented Jan 25, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

priankakariat commented Feb 7, 2024 • edited Loading

priankakariat commented Feb 7, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

priankakariat Feb 7, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

st-tuanmai commented Feb 23, 2024

PaulTR commented Mar 18, 2024

priankakariat commented Mar 18, 2024

PaulTR commented Mar 25, 2024

priankakariat left a comment •

edited

Loading

st-tuanmai commented Jan 10, 2024 •

edited

Loading

st-tuanmai commented Jan 16, 2024 •

edited

Loading

st-tuanmai commented Jan 16, 2024 •

edited

Loading

priankakariat commented Jan 18, 2024 •

edited

Loading

st-tuanmai commented Jan 19, 2024 •

edited

Loading

priankakariat commented Feb 7, 2024 •

edited

Loading

priankakariat Feb 7, 2024 •

edited

Loading