Poor Audio Quality with input_values Input in Parler_TTS

[LiuZH-19]

I am using the Parler_TTS model with a reference audio (input_values) during inference, similar to MusicGen, to perform continuation tasks.

model.generate(input_ids=input_ids, prompt_input_ids=prompt_input_ids, input_values=input_values)

While the model continues in the style of the reference audio, the resulting audio quality is poor and contains a lot of noise.
Why does the audio quality degrade when using a reference audio input, and how can this be improved?

Thank you!

[ylacombe]

we should remove input_values as it's not used in this model, it's an artifact left from the fact I was inspired by Musicgen architecture

[stg1205]

change the code after this comment: "# revert the pattern delay mask by filtering the eos and bos token ids from the delay pattern mask"
to

if "input_values" in model_kwargs:
            mask = (output_ids != generation_config.bos_token_id) & (output_ids != generation_config.pad_token_id)
else:
          _, mask = self.decoder.build_delay_pattern_mask(
              input_ids,
              bos_token_id=generation_config.bos_token_id,
              pad_token_id=generation_config.pad_token_id,
              max_length=output_ids.shape[1],
          )
          mask = (mask != generation_config.bos_token_id) & (mask != generation_config.pad_token_id)

I haven't looked into any details, for now it works. I found this bug by comparing the output_ids with the original input_ids encoded by dac, and there are some wrong delays in output_ids.

[Guppy16]

#110

Does this help? I have an example notebook doing continuation as well. You need to use the decode_input_ids argument (as well as fix a bug similar to how @stg1205 showed above)