Question about compute_type.

[despairTK]

I currently use RTX4090 to generate subtitles, but I still don’t understand much about compute_type selection. I also read the introduction here [https://opennmt.net/CTranslate2/quantization.html]. Regarding the computing power of RRTX4090, is the matching compute_type type float32 or other settings?

[Purfview]

Use fastest for you.
Could you test --compute_type=auto --verbose=true and post what type was selected?

[despairTK]

Number of visible GPU devices: 1

Supported compute types by GPU: {'int8_float16', 'int8_float32', 'bfloat16', 'int8_bfloat16', 'int8', 'float32', 'float16'}

[2023-09-30 09:19:56.036] [ctranslate2] [thread 13576] [info] CPU: GenuineIntel (SSE4.1=true, AVX=true, AVX2=true, AVX512=false)
[2023-09-30 09:19:56.036] [ctranslate2] [thread 13576] [info] - Selected ISA: AVX2
[2023-09-30 09:19:56.036] [ctranslate2] [thread 13576] [info] - Use Intel MKL: true
[2023-09-30 09:19:56.036] [ctranslate2] [thread 13576] [info] - SGEMM backend: MKL (packed: false)
[2023-09-30 09:19:56.037] [ctranslate2] [thread 13576] [info] - GEMM_S16 backend: MKL (packed: false)
[2023-09-30 09:19:56.037] [ctranslate2] [thread 13576] [info] - GEMM_S8 backend: MKL (packed: false, u8s8 preferred: true)
[2023-09-30 09:19:56.037] [ctranslate2] [thread 13576] [info] GPU #0: NVIDIA GeForce RTX 4090 (CC=8.9)
[2023-09-30 09:19:56.037] [ctranslate2] [thread 13576] [info] - Allow INT8: true
[2023-09-30 09:19:56.038] [ctranslate2] [thread 13576] [info] - Allow FP16: true (with Tensor Cores: true)
[2023-09-30 09:19:56.038] [ctranslate2] [thread 13576] [info] - Allow BF16: true
[2023-09-30 09:20:05.097] [ctranslate2] [thread 13576] [info] Using CUDA allocator: cuda_malloc_async
[2023-09-30 09:20:05.423] [ctranslate2] [thread 13576] [info] Loaded model C:\Whisper-Faster_models\faster-whisper-large-v2 on device cuda:0
[2023-09-30 09:20:05.424] [ctranslate2] [thread 13576] [info] - Binary version: 6
[2023-09-30 09:20:05.424] [ctranslate2] [thread 13576] [info] - Model specification revision: 3
[2023-09-30 09:20:05.424] [ctranslate2] [thread 13576] [info] - Selected compute type: int8_float16

Model loaded in: 9.45 seconds

According to the test, int8_float16 should be selected?

[despairTK]

Sorry, I didn't express myself clearly. I mean, according to the test results, whisper-faster chose int8_float16. Is this choice the best result that best matches my PC hardware?

[Purfview]

Allegedly auto selects fastest, but I'm skeptical about it.
Better do benchmarks yourself.

[despairTK]

If the subtitle generation results are based on accuracy, is the choice of compute_type also random? Or is one of the options {'int8_float16', 'int8_float32', 'bfloat16', 'int8_bfloat16', 'int8', 'float32', 'float16'} the most accurate?

[Purfview]

Most accurate theoretically can be most inaccurate in practice. :)

Use fastest, that's it.

On different samples different types can be more accurate.