Skip to content

Commit 8bfcf38

Browse files
sguggersgugger
committed
Add doc notebooks with creation script
1 parent 27f3013 commit 8bfcf38

29 files changed

+13016
-0
lines changed

‎Makefile

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
.PHONY: doc-notebooks
2+
3+
doc-notebooks:
4+
python utils/convert_doc_to_notebooks.py

‎docs/benchmarks.ipynb

Lines changed: 463 additions & 0 deletions
Large diffs are not rendered by default.

‎docs/imgs/ppl_chunked.gif

352 KB
Loading

‎docs/imgs/ppl_full.gif

418 KB
Loading

‎docs/imgs/ppl_sliding.gif

373 KB
Loading

‎docs/multilingual.ipynb

Lines changed: 230 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,230 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {},
6+
"source": [
7+
"# Multi-lingual models"
8+
]
9+
},
10+
{
11+
"cell_type": "markdown",
12+
"metadata": {},
13+
"source": [
14+
"Most of the models available in this library are mono-lingual models (English, Chinese and German). A few\n",
15+
"multi-lingual models are available and have a different mechanisms than mono-lingual models.\n",
16+
"This page details the usage of these models.\n",
17+
"\n",
18+
"The two models that currently support multiple languages are BERT and XLM."
19+
]
20+
},
21+
{
22+
"cell_type": "markdown",
23+
"metadata": {},
24+
"source": [
25+
"## XLM"
26+
]
27+
},
28+
{
29+
"cell_type": "markdown",
30+
"metadata": {},
31+
"source": [
32+
"XLM has a total of 10 different checkpoints, only one of which is mono-lingual. The 9 remaining model checkpoints can\n",
33+
"be split in two categories: the checkpoints that make use of language embeddings, and those that don't"
34+
]
35+
},
36+
{
37+
"cell_type": "markdown",
38+
"metadata": {},
39+
"source": [
40+
"### XLM & Language Embeddings"
41+
]
42+
},
43+
{
44+
"cell_type": "markdown",
45+
"metadata": {},
46+
"source": [
47+
"This section concerns the following checkpoints:\n",
48+
"\n",
49+
"- `xlm-mlm-ende-1024` (Masked language modeling, English-German)\n",
50+
"- `xlm-mlm-enfr-1024` (Masked language modeling, English-French)\n",
51+
"- `xlm-mlm-enro-1024` (Masked language modeling, English-Romanian)\n",
52+
"- `xlm-mlm-xnli15-1024` (Masked language modeling, XNLI languages)\n",
53+
"- `xlm-mlm-tlm-xnli15-1024` (Masked language modeling + Translation, XNLI languages)\n",
54+
"- `xlm-clm-enfr-1024` (Causal language modeling, English-French)\n",
55+
"- `xlm-clm-ende-1024` (Causal language modeling, English-German)\n",
56+
"\n",
57+
"These checkpoints require language embeddings that will specify the language used at inference time. These language\n",
58+
"embeddings are represented as a tensor that is of the same shape as the input ids passed to the model. The values in\n",
59+
"these tensors depend on the language used and are identifiable using the `lang2id` and `id2lang` attributes\n",
60+
"from the tokenizer.\n",
61+
"\n",
62+
"Here is an example using the `xlm-clm-enfr-1024` checkpoint (Causal language modeling, English-French):"
63+
]
64+
},
65+
{
66+
"cell_type": "code",
67+
"execution_count": null,
68+
"metadata": {},
69+
"outputs": [],
70+
"source": [
71+
"import torch\n",
72+
"from transformers import XLMTokenizer, XLMWithLMHeadModel\n",
73+
"tokenizer = XLMTokenizer.from_pretrained(\"xlm-clm-enfr-1024\")\n",
74+
"model = XLMWithLMHeadModel.from_pretrained(\"xlm-clm-enfr-1024\")"
75+
]
76+
},
77+
{
78+
"cell_type": "markdown",
79+
"metadata": {},
80+
"source": [
81+
"The different languages this model/tokenizer handles, as well as the ids of these languages are visible using the\n",
82+
"`lang2id` attribute:"
83+
]
84+
},
85+
{
86+
"cell_type": "code",
87+
"execution_count": null,
88+
"metadata": {},
89+
"outputs": [
90+
{
91+
"data": {
92+
"text/plain": [
93+
"{'en': 0, 'fr': 1}"
94+
]
95+
},
96+
"execution_count": null,
97+
"metadata": {},
98+
"output_type": "execute_result"
99+
}
100+
],
101+
"source": [
102+
"print(tokenizer.lang2id)"
103+
]
104+
},
105+
{
106+
"cell_type": "markdown",
107+
"metadata": {},
108+
"source": [
109+
"These ids should be used when passing a language parameter during a model pass. Let's define our inputs:"
110+
]
111+
},
112+
{
113+
"cell_type": "code",
114+
"execution_count": null,
115+
"metadata": {},
116+
"outputs": [],
117+
"source": [
118+
"input_ids = torch.tensor([tokenizer.encode(\"Wikipedia was used to\")]) # batch size of 1"
119+
]
120+
},
121+
{
122+
"cell_type": "markdown",
123+
"metadata": {},
124+
"source": [
125+
"We should now define the language embedding by using the previously defined language id. We want to create a tensor\n",
126+
"filled with the appropriate language ids, of the same size as input_ids. For english, the id is 0:"
127+
]
128+
},
129+
{
130+
"cell_type": "code",
131+
"execution_count": null,
132+
"metadata": {},
133+
"outputs": [],
134+
"source": [
135+
"language_id = tokenizer.lang2id['en'] # 0\n",
136+
"langs = torch.tensor([language_id] * input_ids.shape[1]) # torch.tensor([0, 0, 0, ..., 0])\n",
137+
"# We reshape it to be of size (batch_size, sequence_length)\n",
138+
"langs = langs.view(1, -1) # is now of shape [1, sequence_length] (we have a batch size of 1)"
139+
]
140+
},
141+
{
142+
"cell_type": "markdown",
143+
"metadata": {},
144+
"source": [
145+
"You can then feed it all as input to your model:"
146+
]
147+
},
148+
{
149+
"cell_type": "code",
150+
"execution_count": null,
151+
"metadata": {},
152+
"outputs": [],
153+
"source": [
154+
"outputs = model(input_ids, langs=langs)"
155+
]
156+
},
157+
{
158+
"cell_type": "markdown",
159+
"metadata": {},
160+
"source": [
161+
"The example [run_generation.py](https://github.com/huggingface/transformers/blob/master/examples/text-generation/run_generation.py)\n",
162+
"can generate text using the CLM checkpoints from XLM, using the language embeddings."
163+
]
164+
},
165+
{
166+
"cell_type": "markdown",
167+
"metadata": {},
168+
"source": [
169+
"### XLM without Language Embeddings"
170+
]
171+
},
172+
{
173+
"cell_type": "markdown",
174+
"metadata": {},
175+
"source": [
176+
"This section concerns the following checkpoints:\n",
177+
"\n",
178+
"- `xlm-mlm-17-1280` (Masked language modeling, 17 languages)\n",
179+
"- `xlm-mlm-100-1280` (Masked language modeling, 100 languages)\n",
180+
"\n",
181+
"These checkpoints do not require language embeddings at inference time. These models are used to have generic\n",
182+
"sentence representations, differently from previously-mentioned XLM checkpoints."
183+
]
184+
},
185+
{
186+
"cell_type": "markdown",
187+
"metadata": {},
188+
"source": [
189+
"## BERT"
190+
]
191+
},
192+
{
193+
"cell_type": "markdown",
194+
"metadata": {},
195+
"source": [
196+
"BERT has two checkpoints that can be used for multi-lingual tasks:\n",
197+
"\n",
198+
"- `bert-base-multilingual-uncased` (Masked language modeling + Next sentence prediction, 102 languages)\n",
199+
"- `bert-base-multilingual-cased` (Masked language modeling + Next sentence prediction, 104 languages)\n",
200+
"\n",
201+
"These checkpoints do not require language embeddings at inference time. They should identify the language\n",
202+
"used in the context and infer accordingly."
203+
]
204+
},
205+
{
206+
"cell_type": "markdown",
207+
"metadata": {},
208+
"source": [
209+
"## XLM-RoBERTa"
210+
]
211+
},
212+
{
213+
"cell_type": "markdown",
214+
"metadata": {},
215+
"source": [
216+
"XLM-RoBERTa was trained on 2.5TB of newly created clean CommonCrawl data in 100 languages. It provides strong\n",
217+
"gains over previously released multi-lingual models like mBERT or XLM on downstream taks like classification,\n",
218+
"sequence labeling and question answering.\n",
219+
"\n",
220+
"Two XLM-RoBERTa checkpoints can be used for multi-lingual tasks:\n",
221+
"\n",
222+
"- `xlm-roberta-base` (Masked language modeling, 100 languages)\n",
223+
"- `xlm-roberta-large` (Masked language modeling, 100 languages)"
224+
]
225+
}
226+
],
227+
"metadata": {},
228+
"nbformat": 4,
229+
"nbformat_minor": 4
230+
}

0 commit comments

Comments
 (0)