[safari] Voicing/SpeechSynthesis is buggy in multiple iframes #92

zepumph · 2022-10-19T15:56:20Z

@Nancy-Salpepi reported this bug originally in https://github.com/phetsims/phet-io-wrappers/issues/459. We are finding that our voicing in our sims behaves quite badly when running multiple sims in iframes on the same page in safari.

Test 1:

<iframe src="https://phet-dev.colorado.edu/html/ratio-and-proportion/1.2.0-dev.31/phet/ratio-and-proportion_all_phet.html?voicingInitiallyEnabled&preferencesStorage"></iframe>
<iframe src="https://phet-dev.colorado.edu/html/ratio-and-proportion/1.2.0-dev.31/phet/ratio-and-proportion_all_phet.html?voicingInitiallyEnabled&preferencesStorage"></iframe>
<iframe src="https://phet-dev.colorado.edu/html/ratio-and-proportion/1.2.0-dev.31/phet/ratio-and-proportion_all_phet.html?voicingInitiallyEnabled&preferencesStorage"></iframe>

This consistently showed that the last iframe's voicing worked perfectly, but the other two were silent. Behavior is great on chrome, with all three iframes working.

Then I added @marlitas and @samreid (our resident safari devices holders) and we made this test:

iframe contents:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="utf-8"/>
  <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"/>
  <meta name="viewport" content="width=device-width"/>

  <title>Speech synthesiser</title>

  <link rel="stylesheet" href="style.css"/>
</head>

<body>
<h1>Speech synthesiser</h1>

<p>
  Enter some text in the input below and press return or the "play" button
  to hear it. change voices using the dropdown menu.
</p>

<form>
  <label for="txt">Enter text</label>
  <input id="txt" type="text" class="txt"/>
  <div>
    <label for="rate">Rate</label
    ><input type="range" min="0.5" max="2" value="1" step="0.1" id="rate"/>
    <div class="rate-value">1</div>
    <div class="clearfix"></div>
  </div>
  <div>
    <label for="pitch">Pitch</label
    ><input type="range" min="0" max="2" value="1" step="0.1" id="pitch"/>
    <div class="pitch-value">1</div>
    <div class="clearfix"></div>
  </div>
  <select></select>
  <div class="controls">
    <button id="play" type="submit">Play</button>
  </div>
</form>

<script>
  const synth = window.speechSynthesis;

  const inputForm = document.querySelector( "form" );
  const inputTxt = document.querySelector( ".txt" );
  const voiceSelect = document.querySelector( "select" );

  const pitch = document.querySelector( "#pitch" );
  const pitchValue = document.querySelector( ".pitch-value" );
  const rate = document.querySelector( "#rate" );
  const rateValue = document.querySelector( ".rate-value" );

  let voices = [];

  function populateVoiceList() {
    voices = synth.getVoices().sort( function( a, b ) {
      const aname = a.name.toUpperCase();
      const bname = b.name.toUpperCase();

      if ( aname < bname ) {
        return -1;
      }
      else if ( aname == bname ) {
        return 0;
      }
      else {
        return +1;
      }
    } );
    const selectedIndex =
      voiceSelect.selectedIndex < 0 ? 0 : voiceSelect.selectedIndex;
    voiceSelect.innerHTML = "";

    for ( let i = 0; i < voices.length; i++ ) {
      const option = document.createElement( "option" );
      option.textContent = `${voices[ i ].name} (${voices[ i ].lang})`;

      if ( voices[ i ].default ) {
        option.textContent += " -- DEFAULT";
      }

      option.setAttribute( "data-lang", voices[ i ].lang );
      option.setAttribute( "data-name", voices[ i ].name );
      voiceSelect.appendChild( option );
    }
    voiceSelect.selectedIndex = selectedIndex;
  }

  populateVoiceList();

  if ( speechSynthesis.onvoiceschanged !== undefined ) {
    speechSynthesis.onvoiceschanged = populateVoiceList;
  }

  function speak() {
    // if ( synth.speaking ) {
    //   console.error( "speechSynthesis.speaking" );
    //   return;
    // }

    if ( inputTxt.value !== "" ) {
      const utterThis = new SpeechSynthesisUtterance( inputTxt.value );

      utterThis.onend = function( event ) {
        console.log( "SpeechSynthesisUtterance.onend" );
      };

      utterThis.onerror = function( event ) {
        console.error( "SpeechSynthesisUtterance.onerror" );
      };

      const selectedOption =
        voiceSelect.selectedOptions[ 0 ].getAttribute( "data-name" );

      for ( let i = 0; i < voices.length; i++ ) {
        if ( voices[ i ].name === selectedOption ) {
          utterThis.voice = voices[ i ];
          break;
        }
      }
      utterThis.pitch = pitch.value;
      utterThis.rate = rate.value;
      console.log( inputTxt.value );
      synth.speak( utterThis );
    }
  }

  inputForm.onsubmit = function( event ) {
    event.preventDefault();

    let count = 0;
    const id = setInterval( () => {
      speak();
      count++;

      if ( count >= 5 ) {
        clearInterval( id );
      }
    }, 1000 );

    inputTxt.blur();
  };

  pitch.onchange = function() {
    pitchValue.textContent = pitch.value;
  };

  rate.onchange = function() {
    rateValue.textContent = rate.value;
  };

  voiceSelect.onchange = function() {
    speak();
  };
</script>
</body>
</html>

outer webpage:

<html>
<body>
<iframe src="html.html" width="800" height="400"></iframe>
<iframe src="html.html" width="800" height="400"></iframe>
</body>
</html>

In that we got semi sporadic behavior where it is possible that when only interacting with iframe 1, we hear a single voiced alert and then never hear anything again.

It is also possible that we hear everything from iframe 1, but then when go to use the second iframe, we hear just a single voiced alert, and then never anything again. We will need to investigate more, but I believe this should block voicing sims until we learn more about it.

zepumph · 2022-10-19T15:56:28Z

Tagging @jessegreenberg.

jessegreenberg · 2022-10-20T13:04:49Z

Bummer. Did the problem only present itself when changing the voice and its rate and pitch? I am wondering if it would happen if html.html is reduced to a button whose click listener is just

window.speechSynthesis.speak( new SpeechSynthesisUtterance( 'This is a test' ) );

zepumph · 2022-11-09T15:51:59Z

@emily-phet and I spoke about this today and don't think that we should investigate this right now. Unmarking as blocking.

Nancy-Salpepi · 2023-01-25T20:29:42Z

Noting this is still an issue using macOS 13.1 and safari 16.2 while testing phetsims/qa#886

zepumph self-assigned this Oct 19, 2022

zepumph added status:blocks-publication dev:voicing labels Oct 19, 2022

zepumph mentioned this issue Oct 19, 2022

Issues blocking friction phetsims/friction#311

Closed

zepumph removed the status:blocks-publication label Nov 9, 2022

zepumph mentioned this issue Nov 9, 2022

What to document about Camera Input: hands phetsims/ratio-and-proportion#512

Closed

zepumph removed their assignment Mar 3, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[safari] Voicing/SpeechSynthesis is buggy in multiple iframes #92

[safari] Voicing/SpeechSynthesis is buggy in multiple iframes #92

zepumph commented Oct 19, 2022

zepumph commented Oct 19, 2022

jessegreenberg commented Oct 20, 2022

zepumph commented Nov 9, 2022

Nancy-Salpepi commented Jan 25, 2023

[safari] Voicing/SpeechSynthesis is buggy in multiple iframes #92

[safari] Voicing/SpeechSynthesis is buggy in multiple iframes #92

Comments

zepumph commented Oct 19, 2022

zepumph commented Oct 19, 2022

jessegreenberg commented Oct 20, 2022

zepumph commented Nov 9, 2022

Nancy-Salpepi commented Jan 25, 2023