Course 5: Agent Observability, Evaluation, and Quality Engineering
------------------------------------------------------------------ What you will learn ------------------- (a) Trace every agent step with OpenTelemetry-compatible tooling (b) Evaluate retrieval and generation separately with DeepEval and RAGAS (c) Write behavioral contracts and generate synthetic test sets from them (d) Use LLM-as-judge and audit it against human raters with Cohen's kappa (e) Gate releases on quality thresholds in CI Why take it ----------- (a) Agents fail silently and intermittently; instinct is not enough (b) Gives you a reusable evaluation harness and shared vocabulary with product and compliance teams What makes it unique -------------------- (a) Treats agent quality as engineering, with CI gates rather than vibe checks (b) Includes a live drift scenario: a defect is seeded, an alert fires, and you walk the trace to the root cause
