This year we will learn some basic deep learning techniques while also talking about some of the fundamental concepts that make them so efficient: simplicity biases, overparametrisation, inductive biases. We will also look into the Transformers, the workhorse of modern language models and more.

A more advance topic we dive into is discrete diffusion-based language modeling. While diffusion has mainly been used for generating images and other continuous data, more recently efforts have been made to apply them to discrete generative AI tasks, such as language models. While the maths behind these models can be pretty complicated to understand even for those with university-level maths knowledge, there are some simpler formulations that may be easier to understand.