[NOT FOR MERGING] Added iOS TFLAudioRecord and GMLAudio in Sound Classification Sample

[priankakariat]

1. Added TFLAudioRecord and GMLAudio which will be used in the task library for audio input.
2. Modified the sample to use these instead of AudioInputManager

[priankakariat]

@khanhlvg Please see my note on why callback is used instead of polling like in Java.
Java AudioRecord read returns immediately as opposed to iOS installTapOnBus which delivers results in completion handler.

[khanhlvg]

We want to support multiple channels model. Both Python and iOS APIs support that.

Multiple channels ByteBuffer can be stored in row-major order in float* array

[priankakariat]

The channels should be interleaved in a flat array. I think that's the way the task library expects the data to be and Java and python works this way.

Also the supported channel count is only internal to GMLAudioRecord. GMLAudio still supports multiple channels. If it's a file source, then we don't use the AudioRecord and it will work fine.

The problem is that currently the only format supported by AVAudioEngine is mono when we request audio samples. iOS 14 upwards there maybe a way to manipulate the avaudio session to connect to stereo. But I am not sure if after doing this it will let us use the audio engine for tapping the stereo mic to get the samples.
We may have to create a full audio library using low level C APIs in iOS to do this.

[khanhlvg]

The channels should be interleaved in a flat array. I think that's the way the task library expects the data to be and Java and python works this way.

Yes that's correct.

The problem is that currently the only format supported by AVAudioEngine is mono when we request audio samples. iOS 14 upwards there maybe a way to manipulate the avaudio session to connect to stereo. But I am not sure if after doing this it will let us use the audio engine for tapping the stereo mic to get the samples.
We may have to create a full audio library using low level C APIs in iOS to do this.

Can you implement it this way to make it consistent with the Python and Android API?

• No need to support 3 or more channels. https://github.com/tensorflow/tflite-support/blob/d6829295cd04284f219e6b08594c699fc9abcf1a/tensorflow_lite_support/java/src/java/org/tensorflow/lite/task/audio/classifier/AudioClassifier.java#L447
• For iOS 13 and lower: Record in mono and duplicate the same content across the two channels if the model requires 2 channels.
• For iOS 14 and later: Record in stero.

[khanhlvg]

Please change to loadFromAudioRecord:(TFLAudioRecord *)audioRecord withError:(NSError **)error to align with the Python and Android APIs. Please also see the comments in TFLAudioRecord.

[priankakariat]

Fixed to match Adroid

[khanhlvg]

Please rename to startRecordingWithError and stop to align with the Python and Android API.

The TFLAudioRecord should have a getter buffer to return the internal buffer, rather than returning the internal buffer only when start recording.

[priankakariat]

The getter will have a completion handler then. Because iOS AudioEngine returns data in callback continuously. Audio callbacks in iOS is always returned in a background thread. So we will have to ensure reads and writes happen in the same thread as reading from a different thread will return out of sync data. For this a dispatch_barrier_async will have to be used which won't return immediately.
So if we cut out the completion handler for data delivery here, we will have a completion handler in read. Also we won't be able to keep track of errors in real time. If there is any error we will have to populate the buffer with -1s or something and leave it to the users to figure out the issue. The read method can't return a status too as read will happen in a callback.

[khanhlvg]

I think what's important is not that reading and writing happens in the same thread, but reading and writing don't happen at the same time. You can use a GCD's serial queue to make sure that read and write to the AudioRecord's internal buffer don't happen at the same time. In the getter, you can use GCD's sync method to in the getter to wait for reading from the internal's queue to complete before returning the copy of the buffer values in the correct order to the caller.

[priankakariat]

I have fixed this according to your suggestion. Do have a look

[khanhlvg]

As this call requires user to add a permission description string to the Info.plist, I think we should let the app developer get the permission in their app, outside of the scope of the Task Library. This will be more aligned to the Android API. (Python doesn't have the concept of permission request).

Could you change the API as follow?

• Check if the app has recording permission.
  • If it's denied, return a "permission denied" NSError.
  • If it's undetermined, return an NSError with the error message suggesting the user to read the Apple's requestRecordPermission documentation and request permission.
• Change the startRecordingWithCompletionHandler method to startRecordingWithError as we don't async interface anymore.

https://developer.apple.com/documentation/avfaudio/avaudiosession/1616463-recordpermission

[priankakariat]

This is fixed now.