Generative AI Language Modeling with Transformers

Generative AI Language Modeling with Transformers

This course is part of multiple programs.

Instructors: Joseph Santarcangelo

Access provided by Inter IKEA

24,470 already enrolled

2 modules

Gain insight into a topic and learn the fundamentals.

140 reviews

Intermediate level

Recommended experience

Flexible schedule

9 hours to complete

Learn at your own pace

91%

Most learners liked this course

2 modules

Gain insight into a topic and learn the fundamentals.

140 reviews

Intermediate level

Recommended experience

Flexible schedule

9 hours to complete

Learn at your own pace

91%

Most learners liked this course

What you'll learn

Explain the role of attention mechanisms in transformer models for capturing contextual relationships in text
Describe the differences in language modeling approaches between decoder-based models like GPT and encoder-based models like BERT
Implement key components of transformer models, including positional encoding, attention mechanisms, and masking, using PyTorch
Apply transformer-based models for real-world NLP tasks, such as text classification and language translation, using PyTorch and Hugging Face tools

Skills you'll gain

Tools you'll learn

Details to know

Shareable certificate

Add to your LinkedIn profile

Assessments

6 assignments

Taught in English

See how employees at top companies are mastering in-demand skills

Learn more about Coursera for Business

logos of Petrobras, TATA, Danone, Capgemini, P&G and L'Oreal

Build your subject-matter expertise

This course is available as part of

When you enroll in this course, you'll also be asked to select a specific program.

Learn new concepts from industry experts
Gain a foundational understanding of a subject or tool
Develop job-relevant skills with hands-on projects
Earn a shareable career certificate

There are 2 modules in this course

This course provides a practical introduction to using transformer-based models for natural language processing (NLP) applications. You will learn to build and train models for text classification using encoder-based architectures like Bidirectional Encoder Representations from Transformers (BERT), and explore core concepts such as positional encoding, word embeddings, and attention mechanisms.

The course covers multi-head attention, self-attention, and causal language modeling with GPT for tasks like text generation and translation. You will gain hands-on experience implementing transformer models in PyTorch, including pretraining strategies such as masked language modeling (MLM) and next sentence prediction (NSP). Through guided labs, you’ll apply encoder and decoder models to real-world scenarios. This course is designed for learners interested in generative AI engineering and requires prior knowledge of Python, PyTorch, and machine learning. Enroll now to build your skills in NLP with transformers!

In this module, you will learn how transformers process sequential data using positional encoding and attention mechanisms. You will explore how to implement positional encoding in PyTorch and understand how attention helps models focus on relevant parts of input sequences. You'll dive deeper into self-attention and scaled dot-product attention with multiple heads to see how they contribute to language modeling tasks. The module also explains how the transformer architecture leverages these mechanisms efficiently. Through hands-on labs, you’ll implement these concepts and build transformer encoder layers in PyTorch. Finally, you'll apply transformer models for text classification, including building a data pipeline, defining the model, and training it, while also exploring techniques to optimize transformer training performance.

What's included

6 videos4 readings2 assignments2 app items2 plugins

6 videos Total 40 minutes

Course Introduction 3 minutes
Positional Encoding 7 minutes
Attention Mechanism 7 minutes
Self-attention Mechanism 7 minutes
From Attention to Transformers 7 minutes
Transformers for Classification: Encoder 9 minutes

4 readings Total 17 minutes

Course Overview 5 minutes
Specialization Overview 7 minutes
Optimization Techniques for Efficient Transformer Training 3 minutes
Summary and Highlights 2 minutes

2 assignments Total 45 minutes

Graded Quiz: Fundamental Concepts of Transformer Architecture 30 minutes
Practice Quiz: Positional Encoding, Attention, and Application in Classification 15 minutes

2 app items Total 105 minutes

Hands-on Lab: Attention Mechanism and Positional Encoding 45 minutes
Hands-on Lab: Applying Transformers for Classification 60 minutes

2 plugins Total 7 minutes

Helpful Tips for Course Completion 2 minutes
Reading: Beginner's Guide to Transformer Model Fundamentals 5 minutes

In this module, you will learn how decoder-based models like GPT are trained using causal language modeling and implemented in PyTorch for both training and inference. You will explore encoder-based models, such as Bidirectional Encoder Representations from Transformers (BERT), and understand their pretraining strategies using masked language modeling (MLM) and next sentence prediction (NSP), along with data preparation techniques in PyTorch. You will also examine how transformer architectures are applied to machine translation, including their implementation using PyTorch. Through hands-on labs, you will gain practical experience with decoder models, encoder models, and translation tasks. The module concludes with a cheat sheet, glossary, and summary to help consolidate your understanding of key concepts.