[Feat]: Estimate tokens per second before downloading

[ChristopherKing42]

Description
Okay, this is very ambitious, but if there was a way to estimate the tokens per second that would be pretty cool. Like, when searching in hugging face each search result has a token per second estimate.

This will probably require a statistical model or even a NN. It would take the phones specs into account.

[a-ghorbani]

This dataset is expanding: https://huggingface.co/spaces/a-ghorbani/ai-phone-leaderboard
We maybe able to utilize it to estimate tg/s and memory usage etc

[ChristopherKing42]

Yeah I'm thinking you have a small model (basically just a statistical model) taking phone specs and LLM specs as input and outputs a normal distribution (i.e. mean and variance) and you optimize the cross entropy.