Long inference time while using flag_gems.use_gems()

[lirui635693142]

Please describe your question

While testing model accuracy, I found that the inference time of flag_gems.use_gems() is extremely higher than that in torch.no_grad().

[iclementine]

The main reason for this is the the Autotuner, which tunes for the best config when it encounters inputs with new key(mainly shape).

We are working on solving this. Possible solutions are

1. Use heuristics instead of Autotuner;
2. use smaller config space;
3. Use the int(log2(size)) instead of size as Autotuner keys;
4. Use an Autotuner with persistent cache.