model config manager class

[Vela-zz]

Implement a model manager class for centralized type model management.

The model manager can manage transformer models used in different metric for different language. The manager support list_model_in_use, set_model_for_metric, reset, load_from_file, and save_to_disk, by using this, I think langcheck can

• a centralized evalution model management
• allow user to use the model they perfered, or let them built the eval pipeline with their own build.
• evaluation version control, allow user save evaluation result align with the models evaluate them for better evaluation experinment management and repeated.
• replace the pydoc in same metric in different language that only has a little differernce with the same one.

As it is a prototype, so I base it on zh-support branch to make it do not influence any other in-progress feature's updating.

It can be used like this,

[yosukehigashi]

Implement a model manager class for centralized type model management.

The model manager can manage transformer models used in different metric for different language. The manager support list_model_in_use, set_model_for_metric, reset, load_from_file, and save_to_disk, by using this, I think langcheck can

• a centralized evalution model management
• allow user to use the model they perfered, or let them built the eval pipeline with their own build.
• evaluation version control, allow user save evaluation result align with the models evaluate them for better evaluation experinment management and repeated.
• replace the pydoc in same metric in different language that only has a little differernce with the same one.

Thank you for writing this up!! I like the idea of a Model Manager, and agree that the properties you listed are all nice to have. I feel like it's not quite a "Model Manager" yet though, since it mostly just manages the model name/path.

Concretely, I think that it would be helpful to make the Model Manager return an actual model, rather than just the model name. Here’s an example of what I have in mind:

# Define various model loaders
def load_sentence_transformer(name):
    return SentenceTransformer(name)

def load_auto_model_for_sequence_classification(name):
    tokenizer = AutoTokenizer.from_pretrained(name)
    model = AutoModelForSequenceClassification.from_pretrained(name)
    return tokenizer, model

# Other loaders

# Model Manager loads a model based on the config
class ModelManager:
    def __init__(self, config):
        # Read config

    def get_model(self, language, metric):
        # NOTE: will also need to ensure that we don't reload models unnecessarily
        model_config = self.config[language][metric]
        if model_config['type'] == 'sentence_transformer':
            return load_sentence_transformer(model_config['name'])
        # Handle other types

    def validate_config(self, language='all', metric='all'):
        # Check that the model(s) can be loaded for the specified language and metric given the config

    # Other methods like set, reset, etc.

A couple additional benefits that I think this could bring

• This makes it easy for us to add validation code. Otherwise, the user will need to actually run the metric to see if their config is valid or not
• This unifies LangCheck code around loading local models

Let me know what you think!

[yosukehigashi]

Thanks for the change, this looks great overall!

I committed a few cleanup changes directly, and made a few minor comments in this initial review. I'll play around with the model manager a bit more and try out the caching / config interface and send another follow-up review

[yosukehigashi]

Thanks for the updates @Vela-zz!! This is looking really good now 😄

(FYI I made a bunch of cleanup commits, docstrings and comments mostly)

[yosukehigashi]

Basically LGTM! Couple last questions, but looks good to merge after that 😄

[yosukehigashi]

LGTM, thanks for all of your work on this @Vela-zz!! I'll merge this to the zh branch once tests pass 🎉