Fix Flux: clip_l support (SD3/3.5 improvements included)

[stduhpf]

Fixes #396

I made the clip backend skip the last text projection if the text_projection tensor doesn't exist This is mathematically equaivalent to replacing the text_projection with the identity matrix (aka torch.eye()).
Also replaced the matrix multiplication with a biasless linear layer when text_projection exists (somehow this changes the outcome).

Flux.1 Schnell (q3_k):

	clip-L	ViT-L-14
master
PR

SD3 2B (q8_0):

Master	PR

SD3.5 Large Turbo (q4_1)

Master	PR

[stduhpf]

If anyone know how this is all supposed to work, feel free to improve the code.

[stduhpf]

I'm pretty sure this is not the way it's supposed to work, so I'm drafting this PR until I figure out how to make it work properly. Right now, I think this is only adding some "noise" the the t5 prompt.

[stduhpf]

Ok I believe i got it now

[stduhpf]

Hard-coding the prompt for clip to always be "Painting, in the style of starry night by Van Gogh", while keeping the same example prompt for T5 ("a lovely cat holding a sign says 'flux.cpp'") now gives this result:

I'm 99.99% certain this PR is now ready for merge.

Patch for hardcoded Clip prompt

diff --git a/conditioner.hpp b/conditioner.hpp
index 8a710d1..75efb08 100644
--- a/conditioner.hpp
+++ b/conditioner.hpp
@@ -1038,9 +1038,10 @@ struct FluxCLIPEmbedder : public Conditioner {
         std::vector<float> t5_weights;
         for (const auto& item : parsed_attention) {
             const std::string& curr_text = item.first;
+            const std::string& curr_text_l = "Painting, in the style of starry night by Van Gogh";
             float curr_weight            = item.second;
 
-            std::vector<int> curr_tokens = clip_l_tokenizer.encode(curr_text, on_new_token_cb);
+            std::vector<int> curr_tokens = clip_l_tokenizer.encode(curr_text_l, on_new_token_cb);
             clip_l_tokens.insert(clip_l_tokens.end(), curr_tokens.begin(), curr_tokens.end());
             clip_l_weights.insert(clip_l_weights.end(), curr_tokens.size(), curr_weight);
 

Edit: I tried this test with my previous attempt at fixing clip (e6314d3) and I got a similar result. It was actually working-ish even though it was definitely not implemented like in the official Flux inference code.

Comparison

(all of those with the same hardcoded prompt for clip_l)

PR (af4f83f)	Previous attempt (d7679c9)	Master

[stduhpf]

@leejet does this look good enough to merge?

[Green-Sky]

extra spaces?

[Green-Sky]

try to fit in the formatting more with the surrounding. ( spaces )

[stduhpf]

This breaks SD3.5 support somehow!

[stduhpf]

Ok it works again now.