3_parameter_efficient_finetuning/notebooks/finetune_sft_peft.ipynb inference error

[rkaunismaa]

In the last notebook cell, when attempting to run inference on the trained model, when we enter the following loop:

for prompt in prompts:
print(f" prompt:\n{prompt}")
print(f" response:\n{test_inference(prompt)}")
print("-" * 50)

the very first prompt fed into 'test_inference(prompt)' generates the following error:

'Input length of input_ids is 31, but max_length is set to 20. This can lead to unexpected behavior. ...'

[rkaunismaa]

The fix is to add a value to max_length in the pipe statement:

Change ...

def test_inference(prompt):
prompt = pipe.tokenizer.apply_chat_template(
[{"role": "user", "content": prompt}],
tokenize=False,
add_generation_prompt=True,
)
outputs = pipe(
prompt,
)
return outputs[0]["generated_text"][len(prompt) :].strip()

To ...

def test_inference(prompt):
prompt = pipe.tokenizer.apply_chat_template(
[{"role": "user", "content": prompt}],
tokenize=False,
add_generation_prompt=True,
)
outputs = pipe(
prompt, max_length=512
)
return outputs[0]["generated_text"][len(prompt) :].strip()