Multiple outputs in sampling

[tonylek]

Is there an option to get multiple outputs when sampling?
For example for specific top_p and temperature I want to return 2 options.

I know I can get multiple options when using beam_search but I need it in sampling setup.

[byshiue]

In such case, you could

1. Pass same requests several time together with different random seeds. Then, you should get different results. This direction is more straightforward.
2. Pass the request first, and then pass it several times next with different random seeds. If you enable the reuse_kv_cache, the time of context phase of the second time would be short and it is recommended.