add performance statistics for image generation

[xufang-lisa]

tickets: CVS-157338

[ilya-lavrenov]

Could you please provide example of such prints?
E.g. for SDXL and FLUX

[likholat]

@xufang-lisa let's add a custom struct ImageGenerationPerfMetrics and collect the metrics there:

struct OPENVINO_GENAI_EXPORTS ImageGenerationPerfMetrics {
    float load_time; // model load time (includes reshape & read_model time)
    float generate_duration; // duration of method generate(...)

    MeanStdPair iteration_duration; // Mean-Std time of one generation iteration
    std::map<std::string, float> encoder_inference_duration; // inference durations for each encoder
    MeanStdPair unet_inference_duration; // inference duration for unet model, should be filled with zeros if we don't have unet
    MeanStdPair transformer_inference_duration; // inference duration for transformer model, should be filled with zeros if we don't have transformer
    float vae_encoder_inference_duration; // inference duration of vae_encoder model, should be filled with zeros if we don't use it
    float vae_decoder_inference_duration; // inference duration of  vae_decoder model
 
    bool m_evaluated = false;
 
    RawImageGenerationPerfMetrics raw_metrics;
};
 
struct OPENVINO_GENAI_EXPORTS RawImageGenerationPerfMetrics {
    std::vector<MicroSeconds> unet_inference_durations; // unet durations for each step
    std::vector<MicroSeconds> transformer_inference_durations; // transformer durations for each step
    std::vector<MicroSeconds> iteration_durations;  //  durations of each step
};

I'd also like to propose return ov::Tensor as a generate method output and add get_perfomance_metrics() method

[xufang-lisa]

@xufang-lisa let's add a custom struct ImageGenerationPerfMetrics and collect the metrics there:

updated

[ilya-lavrenov]

@xufang-lisa let's add a custom struct ImageGenerationPerfMetrics and collect the metrics there:

updated

could you please add benchmark for image generation similar to benchmark for GenAI LLM, which can print all these detailed statistic?
See https://github.com/openvinotoolkit/openvino.genai/blob/master/samples/cpp/benchmark_genai/benchmark_genai.cpp#L54-L61

And provide here an output of such sample.

BTW, original samples should be kept as is and avoid printed timings.

[xufang-lisa]

@xufang-lisa let's add a custom struct ImageGenerationPerfMetrics and collect the metrics there:

updated

could you please add benchmark for image generation similar to benchmark for GenAI LLM, which can print all these detailed statistic? See https://github.com/openvinotoolkit/openvino.genai/blob/master/samples/cpp/benchmark_genai/benchmark_genai.cpp#L54-L61

And provide here an output of such sample.

BTW, original samples should be kept as is and avoid printed timings.

@ilya-lavrenov Do you mean to add support for image generation in benchmark_genai?

[ilya-lavrenov]

@xufang-lisa let's add a custom struct ImageGenerationPerfMetrics and collect the metrics there:

updated

could you please add benchmark for image generation similar to benchmark for GenAI LLM, which can print all these detailed statistic? See https://github.com/openvinotoolkit/openvino.genai/blob/master/samples/cpp/benchmark_genai/benchmark_genai.cpp#L54-L61
And provide here an output of such sample.
BTW, original samples should be kept as is and avoid printed timings.

@ilya-lavrenov Do you mean to add support for image generation in benchmark_genai?

no, create dedicated benchmark application for image generation, which is similar to VLM / LLM benchmarks

[ilya-lavrenov]

please, don't imply that model by default is located in current folder.
I would make this parameters as required as in other benchmark applications:

openvino.genai/samples/cpp/text_generation/benchmark_genai.cpp

Lines 10 to 11 in 4fb48de

	options.add_options()
	("m,model", "Path to model and tokenizers base directory", cxxopts::value<std::string>())

[ilya-lavrenov]

can we print more information? E.g. how much time is taken by text encoders, VAE encode / decode, first / other iterations for main denoising loop

Because currently printed information is non informative (about what happens in pipeline) and can be achieved by external benchmarking around generate method.

[ilya-lavrenov]

can we generalize this benchmark to support inpainting / image to image as well?
We can add an argument to specify pipeline type

[ilya-lavrenov]

can external code (within Image generation pipeline) measure time of decode function? W/ such approach we don't need extra output arguments for this method.

The same approach for all other models within this image_generation/models folder.

[ilya-lavrenov]

this field should be hidden inside std::shared_ptr<DiffusionPipeline> m_impl;

[ilya-lavrenov]

should it be moved to base DiffusionPipeline class? All derived pipelines will inherit this field.