The field of Natural Language Processing (NLP) has witnessed rapid progress in recent times, driven mainly by the design and development of Large Language Models (LLMs). With the increase in scale, LLMs exhibit various emergent properties, though there are conflicting opinions among researchers about these phenomena. Nonetheless, LLMs are proving to be useful and are becoming ubiquitous across numerous applications.
This advanced course aims to introduce the latest advancements in generative AI for text and is open to both undergraduate and graduate students. The course is structured into five modules: Basics, Architecture, Learnability, User Acceptability, and Ethics & Miscellaneous. Together, these modules will provide a comprehensive view of the different facets of LLMs.
Students should have a background in Machine Learning and be proficient in Python programming. At least some basic knowledge of Deep Learning and NLP is preferred. Through assignments and a course project, students will acquire the skills necessary to design, implement, and understand LLMs using the PyTorch framework.
• Data Structures and Algorithms
• Python Programming
• Machine Learning
• Deep Learning
• Natural Language Processing
• Course Introduction
• Introduction to Natural Language Processing
• Introduction to Language Models
• Word Representation
• Neural Language Models
• Introduction to Transformer
• Pre-training Strategies
• Advanced Attention Mechanisms
• Mixture of Experts
• Scaling Laws
• Instruction Fine-tuning and In-context Learning
• Alignment
• Efficient Adaptation of LLMs
• Efficient Inference from LLMs
• Retrieval Augmentation
• Tool Augmentation
• Reasoning
• Vision Language Models
• Long-context LLMs
• Model Editing
• Bias, Toxicity and Hallucination
• Self-evolving LLMs
• Interpreting the Inner Workings of LLMs
• Conclusion
Tanmoy Chakraborty
Associate Professor
Office: 3B-7 (Block III, 3rd Floor),
Department of Electrical Engineering,
Indian Institute of Technology Delhi
IIT Delhi Main Rd, IIT Campus, Hauz Khas, New Delhi, Delhi 110016
E-Mail: tanchak@iitd.ac.in; chak.tanmoy.iit@gmail.com