Natural language processing is ubiquitous in modern intelligent technologies, serving as a foundation for language translators, virtual assistants, search engines, and many more. In this course, we cover the foundations of modern methods for natural language processing, such as word embeddings, recurrent neural networks, transformers, and pretraining, and how they can be applied to important tasks in the field, such as machine translation and text classification. We also cover issues with these state-of-the-art approaches (such as robustness, interpretability, sensitivity), identify their failure modes in different NLP applications, and discuss analysis and mitigation techniques for these issues.
| Platform | Where & when |
|---|---|
| Lectures | Wednesdays: 11:15-13:00 [STCC - Cloud C] & Thursdays: 13:15-14:00 [CE16] |
| Exercises Session | Thursdays: 14:15-16:00 [CE11] |
| Project Assistance (not every week) |
Wednesdays: 13:15-14:00 [STCC - Cloud C] |
| QA Forum & Annoucements | Ed Forum [link] |
| Grades | Moodle [link] |
All lectures will be given in person and live streamed on Zoom. The link to the Zoom is available on the Ed Forum (pinned post). Beware that, in the event of a technical failure during the lecture, continuing to accompany the lecture live via zoom might not be possible.
Recording of the lectures will be made available on SwitchTube. We will reuse some of last year's recordings and we may record a few new lectures in case of different lecture contents.
| Week | Date | Topic | Instructor |
|---|---|---|---|
| Week 1 | 21 Feb 22 Feb |
Introduction | Building a simple neural classifier Neural LMs: word embeddings [slides] |
Antoine Bosselut |
| Week 2 | 28 Feb 29 Feb |
LM basics | Neural LMs: Fixed Context Models Neural LMs: RNNs, Backpropagation, Vanishing Gradients; LSTMs |
Antoine Bosselut |
| Week 3 | 6 Mar 7 Mar |
Seq2seq + decoding + attention | Transformers Transformers + Greedy Decoding; GPT |
Antoine Bosselut |
| Week 4 | 13 Mar 14 Mar |
Pretraining: ELMo, BERT, MLM, task generality | Transfer Learning: Introduction Pretraining S2S: BART, T5 |
Antoine Bosselut |
| Week 5 | 20 Mar 21 Mar |
Transfer Learning: Dataset Biases Generation: Task |
Antoine Bosselut |
| Week 6 | 27 Mar 28 Mar |
Generation: Decoding and Training Generation: Evaluation |
Antoine Bosselut |
| EASTER BREAK | |||
| Week 7 | 10 Apr 11 Apr |
In-context Learning - GPT-3 + Prompts | Instruction Tuning Project Description |
Antoine Bosselut |
| Week 8 | 17 Apr 18 Apr |
Ethics in NLP: Bias / Fairness and Toxicity, Privacy, Disinformation No class (Project work; A1 Grade Review Session) |
Anna Sotnikova |
| Week 9 | 24 Apr 25 Apr |
Scaling laws | Model Compression No class (Project work; A1 Grade Review Session) |
Antoine Bosselut |
| Week 10 | 1 May 2 May |
Tokenization: BPE, WP, Char-based | Multilingual LMs Guest Lecture: Kayo Yin |
Negar Foroutan Kayo Yin |
| Week 11 | 8 May 9 May |
Syntactic and Semantic Tasks (NER) | Interpretability: BERTology No class (Project work; A2 Grade Review Session) |
Gail Weiss |
| Week 12 | 15 May 16 May |
Reading Comprehension | Retrieval-augmented LMs No class (Project work; A2 Grade Review Session) |
Antoine Bosselut |
| Week 13 | 22 May 23 May |
Multimodality: L & V Looking forward |
Syrielle Montariol Antoine Bosselut |
| Week 14 | 29 May 30 May |
No class (Project work; A3 Grade Review Session) |
| Week | Date | Topic | Instructor |
|---|---|---|---|
| Week 1 | 22 Feb | Setup + Word embeddings [code] | Mete Ismayilzada |
| Week 2 | 29 Feb | Word embeddings review Language and Sequence-to-sequence models |
Mete Ismayilzada Badr AlKhamissi |
| Week 3 | 7 Mar | Language and Sequence-to-sequence models review Attention + Transformers Assignment 1 Q&A |
Badr AlKhamissi Simin Fan |
| Week 4 | 14 Mar | Attention + Transformers review Pretraining and Transfer Learning Pt. 1 Assignment 1 Q&A |
Simin Fan Badr AlKhamissi |
| Week 5 | 21 Mar | Pretraining and Transfer Learning Pt. 1 review Transfer Learning Pt. 2 Assignment 2 Q&A |
Simin Fan |
| Week 6 | 28 Mar | Transfer Learning Pt. 2 review Text Generation Assignment 2 Q&A |
Simin Fan Deniz Bayazit |
| Week 8 | 4 Apr | EASTER BREAK | |
| Week 7 | 11 Apr | Text Generation review In-context Learning Assignment 3 Q&A |
Badr AlKhamissi Deniz Bayazit Mete Ismayilzada |
| Week 9 | 18 Apr | In-context Learning review Assignment 3 Q&A |
Badr AlKhamissi Deniz Bayazit Mete Ismayilzada |
| Week 10 | 25 Apr | Project | TA meetings on-demand |
| Week 11 | 2 May | Project | TA meetings on-demand |
| Week 12 | 9 May | Project Milestone 1 Feedback |
TA meetings on-demand |
| Week 13 | 16 May | Project | TA meetings on-demand |
| Week 14 | 23 May | Project | TA meetings on-demand |
| Week 15 | 30 May | Project Milestone 2 Feedback |
TA meetings on-demand |
- TAs will provide a small discussion over the last week's exercises, answering any questions and explaining the solutions. (10-15mins)
- TAs will present this week's exercise. (5mins)
- Students will be solving this week's exercises and TAs will provide answers and clarification if needed.
Note: Please make sure you have already done the setup prerequisites to run the coding parts of the exercises. You can find the instructions here.
Your grade in the course will be computed according to the following guidelines:
There will be three assignments throughout the course. They will be released and due according to the following schedule:
- Released: 26 February 2024
- Due: 17 March 2024
- Grade released: 14 April 2024
- Grade review sessions: 18 and 25 April 2024
- Released: 18 March 2024
- Due: 7 April 2024
- Grade released: 5 May 2024
- Grade review sessions: 9 and 16 May 2024
- Released: 1 April 2024
- Due: 21 April 2024
- Grade released: 19 May 2024
- Grade review sessions: 29 and 30 May 2024
Assignment releases will be announced on Ed.
The project will be divided into 2 milestones and a final submission. Each milestone will be worth 15% of the final grade with the remaining 30% being allocated to the final report. Each team will be supervised by one of the course TAs or AEs.
More details on the content of the project and the deliverables of each milestone will be released at a later date.
- Due: 5 May 2024
- Due: 26 May 2024
- The final report, code, and date will be due on June 14th. Students are welcome to turn in their materials ahead of time, as soon as the semester ends.
- Due: 14 June 2024
All assignments and milestones are due at 23:59 on their due date. As we understand that circumstances can make it challenging to abide by these due dates, you will receive 6 late days over the course of the semester to be allocated to the assignments and project milestones as you see fit. No further extensions will be granted. The only exception to this rule is for the final report, code, and data. No extensions will be granted beyond June 14th.
Lecturer: Antoine Bosselut
Teaching assistants: Badr AlKhamissi, Deniz Bayazit, Beatriz Borges, Zeming (Eric) Chen, Simin Fan, Negar Foroutan Eghlidi, Silin Gao, Mete Ismayilzada
Please contact us for any organizational questions or questions related to the course content.