New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Be more efficient with corrupted caching #89

Open

ArthurConmy opened this issue Oct 1, 2023 · 0 comments

Owner

ArthurConmy commented Oct 1, 2023

for k in exp.global_cache.corrupted_cache.keys():
    print(k, exp.global_cache.corrupted_cache[k].shape, k in exp.global_cache.online_cache)

returns lots of unnecessary things:

blocks.0.ln1.hook_scale torch.Size([40, 41, 8, 1]) False
blocks.0.ln1.hook_normalized torch.Size([40, 41, 8, 512]) False
...
blocks.0.attn.hook_attn_scores torch.Size([40, 8, 41, 41]) False
blocks.0.attn.hook_pattern torch.Size([40, 8, 41, 41]) False
blocks.0.attn.hook_z torch.Size([40, 41, 8, 64]) False

we should only be caching things that definitely matter. This should save memory and allow us to use bigger models when corrupted_cache_cpu=False

The text was updated successfully, but these errors were encountered:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment