[Bug] HybridCache not subscriptable

[hudson-ai]

Build currently fails due to gemma2's usage of a HybridCache which doesn't support tuple slicing like the friendlier DynamicCache.

"Fixing" this issue (just throwing the cache away...) immediately uncovered another one -- the HybridCache has a maximum size. If we don't set this manually, it is set to the sequence length of the first token sequence the model is called with. Trying to do another forward pass with more tokens leads to exceptions deep down inside of gemma's implementation. Current "fix" is to again... throw the cache away.

Hoping for something more elegant. But I don't think this is too insane for now.

Note: now taking advantage of Cache.crop for cache implementations that support it. This should prevent conversion back and forth from the "legacy" cache format that we previously assumed. (Should fix #986).

[hudson-ai]

Not sure if this is necessary for SlidingWindowCache?

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 31.25000% with 22 lines in your changes missing coverage. Please review.

Project coverage is 63.53%. Comparing base (917fe35) to head (8d17370).

Files with missing lines	Patch %	Lines
guidance/models/transformers/_transformers.py	31.25%	22 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1047      +/-   ##
==========================================
- Coverage   71.96%   63.53%   -8.43%     
==========================================
  Files          63       63              
  Lines        4769     4797      +28     
==========================================
- Hits         3432     3048     -384     
- Misses       1337     1749     +412     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[hudson-ai]

Passing all the model specific tests in the CI Tests workflow. General tests failing due to azurecloud auth issues -- not sure if I am able to rerun with the right perms. But I believe all is fine and dandy with the PR.

[hudson-ai]

Changes since submitting PR:

• When we overflow the size of the cache, reallocate a cache with double the size rather than deleting the old cache, which would cause the next forward pass to only create a cache that is "just big enough".
  • To be seen whether we need to be a bit more conservative here to avoid OOM issues when the sequence length is long.
  • We only do this for Static and Hybrid caches. Other cache types will be deleted until we write implementations for doubling their sizes. We give a warning in this case.
• When we need to reset the cache due to backtracking, we now call Cache.reset if possible in order to avoid reallocating the cache. This also prevents the cache-doubling from resetting.

[hudson-ai]

@paulbkoch any feedback?

[riedgar-ms]

OK with me if it makes the tests work better. But @paulbkoch likely understands the details better

[hudson-ai]

@Harsha-Nori @paulbkoch would appreciate both sets of your eyes on this before merging -- may be necessary to pass CI for build