Tokenizer Error #87
Unanswered
Sharefah-Alghamdi
asked this question in
Q&A
Replies: 1 comment 1 reply
-
just use |
Beta Was this translation helpful? Give feedback.
1 reply
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hi everyone!
Thank you for sharing with us Arabert!
I hope you can help me with the error I got.
I'm trying to use Arabert in my code.
I downloaded the folder for
bert-base-arabertv02
then I use it like this:
pretrained_model='bert-base-arabertv02/tf1_model.tar.gz'
self.tokenizer = AutoTokenizer.from_pretrained(pretrained_model, do_lower_case=False)
but I got this error.
File "/usr/local/lib/python3.7/dist-packages/transformers/models/auto/tokenization_auto.py", line 383, in from_pretrained
config = AutoConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/transformers/models/auto/configuration_auto.py", line 387, in from_pretrained
config_dict, _ = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/transformers/configuration_utils.py", line 441, in get_config_dict
config_dict = cls._dict_from_json_file(resolved_config_file)
File "/usr/local/lib/python3.7/dist-packages/transformers/configuration_utils.py", line 524, in _dict_from_json_file
text = reader.read()
File "/usr/lib/python3.7/codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte
Beta Was this translation helpful? Give feedback.
All reactions