Add generating batch size

[zhuzilin]

The origin implementation was generating samples one by one, which results in low gpu utility for small model. This commit adds a generating-batch-size config to enable batch generation.

Thank you for your time on reviewing this PR :)

[StellaAthena]

I think a better solution is to drop the word train from the name of the existing argument and use the existing argument for generation. Having two seems needlessly confusing.

[zhuzilin]

I think a better solution is to drop the word train from the name of the existing argument and use the existing argument for generation. Having two seems needlessly confusing.

@StellaAthena I found that the train_batch_size arg relates to the train_micro_batch_size_per_gpu, which has something to do with gradient accumulation. And there is a batch_size arg used solely with LM Evaluation Harness:

gpt-neox/eval_tasks/eval_adapter.py

Lines 70 to 72 in 1e346ef

	self.batch_size = batch_size or (
	neox_args.batch_size * self.dp_world_size
	) # default batch size to bs per gpu * dp size

Maybe using the batch_size arg is better?

[StellaAthena]

My suggestion is to rename train_micro_batch_size_per_gpu to micro_batch_size_per_gpu and allow it to fill both the train and the inference roles.

[zhuzilin]

@StellaAthena
Just a double check, you mean rename train_batch_size to batch_size, train_micro_batch_size_per_gpu to micro_batch_size_per_gpu and use micro_batch_size_per_gpu for inference/generation, right?

[StellaAthena]

@StellaAthena Just a double check, you mean rename train_batch_size to batch_size, train_micro_batch_size_per_gpu to micro_batch_size_per_gpu and use micro_batch_size_per_gpu for inference/generation, right?

No, I mean just the second rename. The training batch size and the eval batch size for a fixed microbatch size per gpu are typically different due to gradient accumulation.

[zhuzilin]

@StellaAthena Updated.

[zhuzilin]

@StellaAthena Hi, could you help me to fix the error raised in CI? Thank you!

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}