[feature request] builder to expose {GQA, MHA} selection as argument

[BowenBao]

Currently these are inferred from the combination of other configurations such as device and dtype. It is more flexible for downstream users if this can be selected by choice.

[baijumeswani]

What is the advantage of doing it this way? The current process is to take advantage of the fact that the model builder is aware of the attention operator for a specific device and dtype.

Is this for experimentation purposes? If so, maybe we can expose a extra_options flag to override the default attention operator.

[BowenBao]

Hi @baijumeswani, the idea is to decouple the tie of device/dtype with built attention op. Consider custom eps that implements attention op with dtype not supported in ort cpu/cuda.