[Question] How to control the parallelism way of modules

[mengniwang95]

Hi , I am new to deepspeed and have some questions about how to use it.

I want to apply tensor parallel to a huggingface model, and want to apply row parallelism to one module while column parallelism to another module, how can I realize it with configuration?

My way to parallel:

        ds_inference_kwargs = {
            "dtype": torch_dtype,
            "tensor_parallel": {"tp_size": world_size},
        }
        model = deepspeed.init_inference(model, **ds_inference_kwargs)

Thanks