Batch for image captioning

[rafstahelin]

Is anyone working on recursive folder batch script for captioning stable diffusion captions? Speed needs to be under 2 seconds for 50 tokens max per image. English phrases

[vealocia]

Hi, @rafstahelin!
For your information, we evaluate Qwen-VL on Flickr30K caption dataset (about 1K images) with 8 gpus, a total batch size of 64 (8 images per gpu) and 30 max tokens per image. It costs about 20 seconds for whole process and 1.3 second for a batch of images (8 images).
There are lots of thing you can do to get faster inference, including but not limited to:

• Batch inference.
• Try our Int-4 quantized model. You can get the weights here.

[rafstahelin]

That's great. However none of us use multi GPU's on runpod. Could you point me in the direction of a service to does this? Is there code to use to be able to implement this?

[ShadoWxShinigamI]

#136 Should be able to batch a directory with this. Modify as required

[FangGet]

import this file: qwen_generation_utils.py

1. call make_context for context_token and construct a batch;
2. call model.generate for batch generation;
3. call decode_token to get final response

[trouble-maker007]

@ShadoWxShinigamI I found your batch code is still chat one image with model.chat, not with batch

[matankley]

@rafstahelin Were you able to run a batch successfully ?

[rafstahelin]

didnt try

…

On Tue, Feb 13, 2024 at 12:00 PM Matan Kleyman ***@***.***> wrote: @rafstahelin <https://github.com/rafstahelin> Were you able to run a batch successfully ? — Reply to this email directly, view it on GitHub <#51 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APNXTFWZYYTVDZJSM4P2ZBDYTNBV3AVCNFSM6AAAAAA4IVUPOCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNBRGIYTGOJTGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- thanks best regards, raf