Set use_cache back to True for HF checkpointer #1488
Conversation
| @@ -500,7 +500,12 @@ def tensor_hook( | |||
|
|
|||
| if dist.get_global_rank() == 0: | |||
| log.debug('Saving Hugging Face checkpoint in global rank 0') | |||
|
|
|||
|
|
|||
| if hasattr(original_model.config, 'use_cache'): | |||
There was a problem hiding this comment.
This should not be set on the original model/config because you might continue training after this function is run. I'd be fine setting it for the new model/config that are getting saved out though.
There was a problem hiding this comment.
great point, thanks!
|
@dakinggg seems like some tests are failing now, and if I am reading this correctly, it is because MPT models, by default, have What is the reason behind this choice for MPT models? I haven't been able to find another model in HF-hub that disables the usage of cache in As for the fix for the tests, I assume we can add a patch which checks if the model is MPT, and if yes, don't set |
|
@eldarkurtic Its fine to have it be True since we explicitly set |
|
@dakinggg okay, got it. So it is safe to leave these tests red? |
|
@eldarkurtic well, no. We can't merge with failing tests, but you can update the test so that it passes |
Most HF models have
use_cacheset toTrueby default, which is manually changed toFalsein llm-foundry (most likely due to huggingface/transformers#28056). This PR setsuse_cacheback to True before saving the model with the HF checkpointer.This makes it a bit more convenient to use models trained with llm-foundry, without having to manually edit
config.jsonandgeneration_config.jsonto setuse_cache.