From 9f2a1c1adda9711ce91b6965f744919bd26b6e86 Mon Sep 17 00:00:00 2001 From: Vladislav Antonyuk Date: Sun, 3 Nov 2024 19:45:33 +0200 Subject: [PATCH 1/2] Update SpeechToText docs, Add OfflineSpeechToText --- docs/maui/essentials/speech-to-text.md | 93 ++++++++++++-------------- 1 file changed, 44 insertions(+), 49 deletions(-) diff --git a/docs/maui/essentials/speech-to-text.md b/docs/maui/essentials/speech-to-text.md index c1365685..6fa5e96b 100644 --- a/docs/maui/essentials/speech-to-text.md +++ b/docs/maui/essentials/speech-to-text.md @@ -7,7 +7,7 @@ ms.date: 05/26/2023 # SpeechToText -The `SpeechToText` API provides the ability to convert speech to text. +The `SpeechToText` API provides the ability to convert speech to text using online recognition. For offline recognition, you can use the `OfflineSpeechToText`. ![Screenshot of SpeechText implemented on macOS](../images/essentials/speech-to-text-mac.gif "SpeechToText on macOS") @@ -20,6 +20,8 @@ Add permissions to `AndroidManifest.xml`: ``` +For `OfflineSpeechToText`, Android 33.0 or higher is required. + # [iOS/MacCatalyst](#tab/ios) Add permissions to `Info.plist` @@ -31,6 +33,8 @@ Add permissions to `Info.plist` SpeechToText requires microphone usage ``` +For `OfflineSpeechToText`, iOS 13.0 or higher is required. + # [Windows](#tab/windows) Add permissions to `Package.appxmanifest` @@ -57,36 +61,6 @@ Add permissions to `tizen-manifest.xml`: The `SpeechToText` can be used as follows in C#: -```csharp -async Task Listen(CancellationToken cancellationToken) -{ - var isGranted = await speechToText.RequestPermissions(cancellationToken); - if (!isGranted) - { - await Toast.Make("Permission not granted").Show(CancellationToken.None); - return; - } - - var recognitionResult = await speechToText.ListenAsync( - CultureInfo.GetCultureInfo(Language), - new Progress(partialText => - { - RecognitionText += partialText + " "; - }), cancellationToken); - - if (recognitionResult.IsSuccessful) - { - RecognitionText = recognitionResult.Text; - } - else - { - await Toast.Make(recognitionResult.Exception?.Message ?? "Unable to recognize speech").Show(CancellationToken.None); - } -} -``` - -or using events: - ```csharp async Task StartListening(CancellationToken cancellationToken) { @@ -99,7 +73,7 @@ async Task StartListening(CancellationToken cancellationToken) speechToText.RecognitionResultUpdated += OnRecognitionTextUpdated; speechToText.RecognitionResultCompleted += OnRecognitionTextCompleted; - await speechToText.StartListenAsync(CultureInfo.CurrentCulture, CancellationToken.None); + await speechToText.StartListenAsync(new SpeechToTextOptions { Culture = CultureInfo.CurrentCulture, ShouldReportPartialResults = true }, CancellationToken.None); } async Task StopListening(CancellationToken cancellationToken) @@ -125,24 +99,16 @@ void OnRecognitionTextCompleted(object? sender, SpeechToTextRecognitionResultCom |Method |Description | |---------|---------| | RequestPermissions | Asks for permission. | -| ListenAsync | Starts speech recognition. | | StartListenAsync | Starts the SpeechToText service. (Real time speech recognition results will be surfaced via RecognitionResultUpdated and RecognitionResultCompleted) | | StopListenAsync | Stops the SpeechToText service. (Speech recognition results will be surfaced via RecognitionResultCompleted) | -### SpeechToTextResult - -The result returned from the `ListenAsync` method. This can be used to verify whether the recognition was successful, and also access any exceptions that may have ocurred during the speech recognition. - -#### Properties +## Properties |Property |Type |Description | |---------|---------|---------| -| Text | `string` | The recognized text. | -| Exception | `Exception` | Gets the `Exception` if the speech recognition operation failed. | -| IsSuccessful | `bool` | Gets a value determining whether the operation was successful. | | CurrentState | `SpeechToTextState` | Gets a current listening state. | -#### Events +## Events |EventName |EventArgs |Description | |---------|---------|---------| @@ -150,6 +116,31 @@ The result returned from the `ListenAsync` method. This can be used to verify wh | RecognitionResultCompleted | `SpeechToTextRecognitionResultCompletedEventArgs` | Triggers when SpeechToText has completed. | | StateChanged | `SpeechToTextStateChangedEventArgs` | Triggers when `CurrentState` has changed. | + +### SpeechToTextOptions + +The `SpeechToTextOptions` class provides the ability to configure the speech recognition service. + +#### Properties + +|Property |Type |Description | +|---------|---------|---------| +| Culture | `CultureInfo` | The spoken language to use for speech recognition. | +| ShouldReportPartialResults | `bool` | Gets or sets if include partial results. `True` by default. | + + +### SpeechToTextResult + +The result returned from the `RecognitionResultCompleted` event. This can be used to verify whether the recognition was successful, and also access any exceptions that may have ocurred during the speech recognition. + +#### Properties + +|Property |Type |Description | +|---------|---------|---------| +| Text | `string` | The recognized text. | +| Exception | `Exception` | Gets the `Exception` if the speech recognition operation failed. | +| IsSuccessful | `bool` | Gets a value determining whether the operation was successful. | + #### Methods |Method |Description | @@ -159,6 +150,7 @@ The result returned from the `ListenAsync` method. This can be used to verify wh > [!WARNING] > `EnsureSuccess` will throw an `Exception` if the recognition operation was unsuccessful. + ## Dependency Registration In case you want to inject service, you first need to register it. @@ -174,12 +166,16 @@ public static class MauiProgram .UseMauiApp() .UseMauiCommunityToolkit(); - builder.Services.AddSingleton(SpeechToText.Default); + builder.Services.AddSingleton(SpeechToText.Default); + // For offline recognition + // builder.Services.AddSingleton(OfflineSpeechToText.Default); return builder.Build(); } } ``` +> In case you need to register both `SpeechToText` and `OfflineSpeechToText`, you can use `KeyedService`. + Now you can inject the service like this: ```csharp @@ -202,12 +198,7 @@ public partial class MainPage : ContentPage return; } - var recognitionResult = await speechToText.ListenAsync( - CultureInfo.GetCultureInfo("uk-ua"), - new Progress(), cancellationToken); - - recognitionResult.EnsureSuccess(); - await Toast.Make($"RecognizedText: {recognitionResult.Text}").Show(cancellationToken); + await speechToText.StartListenAsync(new SpeechToTextOptions { Culture = CultureInfo.CurrentCulture, ShouldReportPartialResults = true }, CancellationToken.None); } } ``` @@ -216,6 +207,10 @@ public partial class MainPage : ContentPage You can find an example of `SpeechToText` in action in the [.NET MAUI Community Toolkit Sample Application](https://github.com/CommunityToolkit/Maui/blob/main/samples/CommunityToolkit.Maui.Sample/Pages/Essentials/SpeechToTextPage.xaml). +For Offline recognition, you can use this sample: [.NET MAUI Community Toolkit Sample Application](https://github.com/CommunityToolkit/Maui/blob/main/samples/CommunityToolkit.Maui.Sample/Pages/Essentials/OfflineSpeechToTextPage.xaml) + +```csharp + ## API You can find the source code for `SpeechToText` over on the [.NET MAUI Community Toolkit GitHub repository](https://github.com/CommunityToolkit/Maui/blob/main/src/CommunityToolkit.Maui.Core/Essentials/SpeechToText/ISpeechToText.shared.cs). From f718dda8425ab46a0dbba941dafd42f3ff033e6c Mon Sep 17 00:00:00 2001 From: Vladislav Antonyuk <33021114+VladislavAntonyuk@users.noreply.github.com> Date: Mon, 4 Nov 2024 08:59:37 +0200 Subject: [PATCH 2/2] Update speech-to-text.md --- docs/maui/essentials/speech-to-text.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/maui/essentials/speech-to-text.md b/docs/maui/essentials/speech-to-text.md index 6fa5e96b..612a1eea 100644 --- a/docs/maui/essentials/speech-to-text.md +++ b/docs/maui/essentials/speech-to-text.md @@ -207,9 +207,7 @@ public partial class MainPage : ContentPage You can find an example of `SpeechToText` in action in the [.NET MAUI Community Toolkit Sample Application](https://github.com/CommunityToolkit/Maui/blob/main/samples/CommunityToolkit.Maui.Sample/Pages/Essentials/SpeechToTextPage.xaml). -For Offline recognition, you can use this sample: [.NET MAUI Community Toolkit Sample Application](https://github.com/CommunityToolkit/Maui/blob/main/samples/CommunityToolkit.Maui.Sample/Pages/Essentials/OfflineSpeechToTextPage.xaml) - -```csharp +For Offline recognition, you can use this sample: [.NET MAUI Community Toolkit Sample Application](https://github.com/CommunityToolkit/Maui/blob/main/samples/CommunityToolkit.Maui.Sample/Pages/Essentials/OfflineSpeechToTextPage.xaml). ## API