LoRA support

[11415142513152119]

Would LoRA support need to be implemented on this end or on the backend end?
Right now trying to load them on Ooba's results in either "AttributeError: 'Exllamav2Model' object has no attribute 'dtype'" on ExLlamav2 or "ValueError: Target modules ['k_proj', 'q_proj', 'up_proj', 'o_proj', 'gate_proj', 'down_proj', 'v_proj'] not found in the base model. Please check the target modules and try again." on ExLlamav2_HF.

[turboderp]

ExLlamaV2 doesn't currently support LoRAs. Support is coming soon, though. After that there may need to be some minor changes to TGW to support whatever the API ends up looking like.

[11415142513152119]

Cool, thanks. Should I keep this open for updates?

[turboderp]

Yep. I'll post here when there's progress.

[turboderp]

btw I forgot to post here. LoRAs work now.

[nikshepsvn]

@turboderp qq: I see that exllamav2 supports Multi-Lora, how is that different from https://github.com/punica-ai/punica or https://github.com/S-LoRA/S-LoRA? Would I be able to use exllama to hot-swap LoRA's at request time?

[turboderp]

I have no idea what punica is all about, but you can hot-swap LoRAs, yes. You pass one or more LoRAs as an argument to the forward pass, and then those are applied during inference. The generator functions also take LoRAs as arguments, so you can swap them in and out dynamically.

Just keep in mind that doing it with an existing cache might not produce correct results, since it's still a little unclear how one LoRA would use the keys/values cached from inference with another LoRA. If you're starting a new generation that isn't a concern, of course.