Memory NodeStorage may create very limited historical

[ad-astra-video]

Describe the bug

My understaning is drivers.NodeStorage is set in starter.go on line 1254 to a MemoryDriver with a cache length of 12. I think the current implementation of [capability]_[model] in the AI selector as the session name will be limited to 12 results before cache starts to clear. This could result in ErrNotFound errors when more than 12 requests are done before the 1st request could be downloaded.

Reproduction steps

Send 13 requests to ByteDance/SDXL-Lightning model and try to download first image. Expect to get ErrNotFound.

Expected behaviour

A session pool per request would solve this and limiting inference requests to 12 per request.

Severity

None

Screenshots / Live demo link

No response

OS

None

Running on

None

AI-worker version

No response

Additional context

No response

[rickstaa]

Tracked internally at https://linear.app/livepeer-ai-spe/issue/LIV-270.