1
- # PyTorch RoFormer
2
- 原版Tensorflow权重(https://github.com/ZhuiyiTechnology/roformer )
3
- - [ chinese_roformer_L-12_H-768_A-12.zip] ( https://pan.baidu.com/s/1fiss862YsGCwf2HvU_Jm-g ) (提取码:xy9x)
4
- - [ chinese_roformer_L-6_H-384_A-6.zip] ( https://pan.baidu.com/s/1iIXgZHHCgrYGXVRRSSCVPg ) (提取码:gy97)
5
- - [ chinese_roformer-char_L-12_H-768_A-12.zip] ( https://pan.baidu.com/s/1Q1pq8F4Fsl6bTipUAkqeDQ ) (提取码:bt94)
6
- - [ chinese_roformer-char_L-6_H-384_A-6.zip] ( https://pan.baidu.com/s/1cc281-M0Rsjlwws5phqzbQ ) (提取码:a44c)
7
- - [ chinese_roformer-sim-char_L-12_H-768_A-12.zip] ( https://pan.baidu.com/s/1f1FB288nv1a6jYjsNCordg ) (提取码:2cgz)
8
- - [ chinese_roformer-sim-char_L-6_H-384_A-6.zip] ( https://pan.baidu.com/s/1r0eJ7shGwQ0RzV9BTFFW4g ) (提取码:h68q)
9
-
10
- 已经转化为PyTorch权重
11
- - [ chinese_roformer_small.zip] ( https://pan.baidu.com/s/1Cx7lhtojTyRF61IKHWXEHw ) (提取码:8znw)
12
- - [ chinese_roformer_base.zip] ( https://pan.baidu.com/s/10W5BYDQSeLyajTWjexZeoQ ) (提取码:bimr)
13
- - [ chinese_roformer_char_base.zip] ( https://pan.baidu.com/s/18bgJ1t_1ke0BXq_Xg02qSQ ) (提取码:oqb5)
14
-
15
- ## 安装(代码已经加入到huggingface仓库)
1
+ # PyTorch RoFormer & RoFormer-V2
2
+ RoFormer模型和RoFormer-V2模型
3
+
4
+ ## 更新
5
+ - 2022/03/21 添加` roformer-v2 ` 的权重, 注:必须使用本仓库的代码,不能使用transformers仓库的代码!!!
6
+
7
+ ## 安装(代码已经加入到huggingface仓库),V2版本需要使用本仓库的代码
16
8
transformers v4.7版本已经发布,可以直接安装使用
17
9
``` bash
18
10
pip install -U transformers
19
11
```
12
+
20
13
## 模型权重对照表
21
14
22
- ### 中文模型
15
+ ### 中文模型 roformer-v2
16
+ | huggingface.co | bert4keras |
17
+ | ---------------------------------- | ------------------------------------------------ |
18
+ | [ roformer_v2_chinese_char_small] ( https://huggingface.co/junnyu/roformer_v2_chinese_char_small ) | [ chinese_roformer-v2-char_L-6_H-384_A-6.zip] ( https://pan.baidu.com/s/1huUrC9P60Afggo8AfiUcmA ) (download code:ttn4) |
19
+ | [ roformer_v2_chinese_char_base] ( https://huggingface.co/junnyu/roformer_v2_chinese_char_base ) | [ chinese_roformer-v2-char_L-12_H-768_A-12.zip] ( https://pan.baidu.com/s/1qcnN4LVKVe0-mnHlkN3-6Q ) (download code:pfoh) |
20
+ | [ roformer_v2_chinese_char_large] ( https://huggingface.co/junnyu/roformer_v2_chinese_char_large ) | [ chinese_roformer-v2-char_L-24_H-1024_A-16.zip] ( https://pan.baidu.com/s/1QiJWSZrGxn8vek-8myvL6w ) (download code:npfv) |
21
+
22
+
23
+ ### 中文模型 roformer-v1
23
24
| huggingface.co | bert4keras |
24
25
| ---------------------------------- | ------------------------------------------------ |
25
26
| [ roformer_chinese_base] ( https://huggingface.co/junnyu/roformer_chinese_base ) | [ chinese_roformer_L-12_H-768_A-12.zip] ( https://pan.baidu.com/s/1fiss862YsGCwf2HvU_Jm-g ) (download code:xy9x) |
@@ -38,34 +39,69 @@ pip install -U transformers
38
39
| [ roformer_small_generator] ( https://huggingface.co/junnyu/roformer_small_generator ) |
39
40
| [ roformer_small_discriminator] ( https://huggingface.co/junnyu/roformer_small_discriminator ) |
40
41
41
-
42
- ## 使用
42
+ ## roformer-v2 MLM测试
43
43
``` python
44
44
import torch
45
- from transformers import RoFormerModel, RoFormerTokenizer, TFRoFormerModel
46
- tokenizer = RoFormerTokenizer.from_pretrained(" junnyu/roformer_chinese_base" )
47
- pt_model = RoFormerModel.from_pretrained(" junnyu/roformer_chinese_base" )
48
- tf_model = TFRoFormerModel.from_pretrained(" junnyu/roformer_chinese_base" ,
49
- from_pt = True )
50
- text = " 这里基本保留了唐宋遗留下来的坊巷格局和大量明清古建筑,其中各级文保单位29处,被誉为“里坊制度的活化石”“明清建筑博物馆”!"
45
+ import tensorflow as tf
46
+ from transformers import BertTokenizer
47
+ from roformer import RoFormerForMaskedLM, TFRoFormerForMaskedLM
48
+
49
+ text = " 今天[MASK]很好,我[MASK]去公园玩。"
50
+ tokenizer = BertTokenizer.from_pretrained(" junnyu/roformer_v2_chinese_char_base" )
51
+ pt_model = RoFormerForMaskedLM.from_pretrained(" junnyu/roformer_v2_chinese_char_base" )
52
+ tf_model = TFRoFormerForMaskedLM.from_pretrained(
53
+ " junnyu/roformer_v2_chinese_char_base" , from_pt = True
54
+ )
51
55
pt_inputs = tokenizer(text, return_tensors = " pt" )
52
56
tf_inputs = tokenizer(text, return_tensors = " tf" )
57
+ # pytorch
53
58
with torch.no_grad():
54
- pt_outputs = pt_model(** pt_inputs).last_hidden_state
55
- print (pt_outputs.shape)
56
- tf_outputs = tf_model(** tf_inputs, training = False ).last_hidden_state
57
- print (tf_outputs.shape)
59
+ pt_outputs = pt_model(** pt_inputs).logits[0 ]
60
+ pt_outputs_sentence = " pytorch: "
61
+ for i, id in enumerate (tokenizer.encode(text)):
62
+ if id == tokenizer.mask_token_id:
63
+ tokens = tokenizer.convert_ids_to_tokens(pt_outputs[i].topk(k = 5 )[1 ])
64
+ pt_outputs_sentence += " [" + " ||" .join(tokens) + " ]"
65
+ else :
66
+ pt_outputs_sentence += " " .join(
67
+ tokenizer.convert_ids_to_tokens([id ], skip_special_tokens = True )
68
+ )
69
+ print (pt_outputs_sentence)
70
+ # tf
71
+ tf_outputs = tf_model(** tf_inputs, training = False ).logits[0 ]
72
+ tf_outputs_sentence = " tf: "
73
+ for i, id in enumerate (tokenizer.encode(text)):
74
+ if id == tokenizer.mask_token_id:
75
+ tokens = tokenizer.convert_ids_to_tokens(tf.math.top_k(tf_outputs[i], k = 5 )[1 ])
76
+ tf_outputs_sentence += " [" + " ||" .join(tokens) + " ]"
77
+ else :
78
+ tf_outputs_sentence += " " .join(
79
+ tokenizer.convert_ids_to_tokens([id ], skip_special_tokens = True )
80
+ )
81
+ print (tf_outputs_sentence)
82
+ # small
83
+ # pytorch: 今天[的||,||是||很||也]很好,我[要||会||是||想||在]去公园玩。
84
+ # tf: 今天[的||,||是||很||也]很好,我[要||会||是||想||在]去公园玩。
85
+ # base
86
+ # pytorch: 今天[我||天||晴||园||玩]很好,我[想||要||会||就||带]去公园玩。
87
+ # tf: 今天[我||天||晴||园||玩]很好,我[想||要||会||就||带]去公园玩。
88
+ # large
89
+ # pytorch: 今天[天||气||我||空||阳]很好,我[又||想||会||就||爱]去公园玩。
90
+ # tf: 今天[天||气||我||空||阳]很好,我[又||想||会||就||爱]去公园玩。
58
91
```
59
- ## MLM测试
92
+
93
+ ## roformer-v1 MLM测试
60
94
``` python
61
95
import torch
62
96
import tensorflow as tf
63
97
from transformers import RoFormerForMaskedLM, RoFormerTokenizer, TFRoFormerForMaskedLM
98
+
64
99
text = " 今天[MASK]很好,我[MASK]去公园玩。"
65
100
tokenizer = RoFormerTokenizer.from_pretrained(" junnyu/roformer_chinese_base" )
66
101
pt_model = RoFormerForMaskedLM.from_pretrained(" junnyu/roformer_chinese_base" )
67
102
tf_model = TFRoFormerForMaskedLM.from_pretrained(
68
- " junnyu/roformer_chinese_base" , from_pt = True )
103
+ " junnyu/roformer_chinese_base" , from_pt = True
104
+ )
69
105
pt_inputs = tokenizer(text, return_tensors = " pt" )
70
106
tf_inputs = tokenizer(text, return_tensors = " tf" )
71
107
# pytorch
@@ -78,22 +114,24 @@ for i, id in enumerate(tokenizer.encode(text)):
78
114
pt_outputs_sentence += " [" + " ||" .join(tokens) + " ]"
79
115
else :
80
116
pt_outputs_sentence += " " .join(
81
- tokenizer.convert_ids_to_tokens([id ], skip_special_tokens = True ))
117
+ tokenizer.convert_ids_to_tokens([id ], skip_special_tokens = True )
118
+ )
82
119
print (pt_outputs_sentence)
83
120
# tf
84
121
tf_outputs = tf_model(** tf_inputs, training = False ).logits[0 ]
85
122
tf_outputs_sentence = " tf: "
86
123
for i, id in enumerate (tokenizer.encode(text)):
87
124
if id == tokenizer.mask_token_id:
88
- tokens = tokenizer.convert_ids_to_tokens(
89
- tf.math.top_k(tf_outputs[i], k = 5 )[1 ])
125
+ tokens = tokenizer.convert_ids_to_tokens(tf.math.top_k(tf_outputs[i], k = 5 )[1 ])
90
126
tf_outputs_sentence += " [" + " ||" .join(tokens) + " ]"
91
127
else :
92
128
tf_outputs_sentence += " " .join(
93
- tokenizer.convert_ids_to_tokens([id ], skip_special_tokens = True ))
129
+ tokenizer.convert_ids_to_tokens([id ], skip_special_tokens = True )
130
+ )
94
131
print (tf_outputs_sentence)
95
132
# pytorch: 今天[天气||天||心情||阳光||空气]很好,我[想||要||打算||准备||喜欢]去公园玩。
96
133
# tf: 今天[天气||天||心情||阳光||空气]很好,我[想||要||打算||准备||喜欢]去公园玩。
134
+
97
135
```
98
136
99
137
## 手动权重转换
0 commit comments