Evaluate & Optimize LLM Performance

This course is part of LLM Optimization & Evaluation Specialization

Instructor: LearningMate

Access provided by Albertsons

3 modules

Gain insight into a topic and learn the fundamentals.

Intermediate level

Recommended experience

4 hours to complete

Flexible schedule

Learn at your own pace

3 modules

Gain insight into a topic and learn the fundamentals.

Intermediate level

Recommended experience

4 hours to complete

Flexible schedule

Learn at your own pace

What you'll learn

Evaluate LLMs using metrics like BLEU & ROUGE run A/B tests for statistical significance, and optimize model performance with data-driven strategies.

Skills you'll gain

Tools you'll learn

Prompt Engineering

Details to know

Shareable certificate

Add to your LinkedIn profile

Assessments

3 assignments¹

AI Graded see disclaimer

Taught in English

See how employees at top companies are mastering in-demand skills

Learn more about Coursera for Business

logos of Petrobras, TATA, Danone, Capgemini, P&G and L'Oreal

Build your subject-matter expertise

This course is part of the LLM Optimization & Evaluation Specialization

When you enroll in this course, you'll also be enrolled in this Specialization.

Learn new concepts from industry experts
Gain a foundational understanding of a subject or tool
Develop job-relevant skills with hands-on projects
Earn a shareable career certificate

There are 3 modules in this course

You've integrated a powerful Large Language Model (LLM) into your application. The initial results are impressive, and your team is excited. But then the hard questions start. Is the new prompt really better than the old one, or does it just "feel" better? How do you prove to stakeholders that switching from GPT-3.5 to GPT-4 is worth the extra cost? When you have two models that give slightly different answers, how do you decide which one is objectively superior?

This introductory module lays the groundwork for quantitative Large Language Mode (LLM) evaluation. Learners will discover why relying on intuition to judge model performance is unsustainable and explore the foundational metrics used to create automated, objective evaluation systems. We will cover both lexical similarity metrics (like BLEU and ROUGE-L) that assess text structure and semantic metrics (like cosine similarity) that capture meaning. By the end of this module, learners will have the conceptual understanding and practical code to build their first automated evaluation script.

What's included

2 videos1 reading1 assignment1 ungraded lab

2 videos Total 12 minutes

How to Compute Lexical Metrics: BLEU & ROUGE-L in Python? 6 minutes
How to Compute Semantic Similarity with Embeddings? 6 minutes

1 reading Total 5 minutes

A Guide to LLM Evaluation: Lexical and Semantic Metrics 5 minutes

1 assignment Total 10 minutes

Knowledge Check: Choosing Your Metrics 10 minutes

1 ungraded lab Total 60 minutes

Building Your First Automated Evaluation Script 60 minutes

This module transitions from raw metrics to credible conclusions. Learners will discover why statistical rigor is non-negotiable when comparing LLM outputs. They will learn to formulate clear hypotheses, design and analyze A/B tests, and interpret results such as p-values and confidence intervals to distinguish true performance gains from random noise. By the end of this module, learners will be equipped to make data-driven decisions with confidence, ensuring that changes to prompts, models, or parameters lead to statistically significant improvements.