About batch size > 1

[Nero0113]

First of all, thank you for open-sourcing the implementation of speculative decoding at batch size > 1. I would like to ask if it is possible to adapt directly to the models downloaded by huggingface instead of customizing their framework code. Because I tried to use this with codegen, but the generated content is messy. Hope you can answer my confusion at your convenience.

[deafTim]

I am also interested in this.

[deafTim]

@Nero0113 what model did you use? So it is possible to use some pre-trained model?