AI Evals For Engineers & PMs

Expert-led course
Designed for both individuals and teams
Hamel Husain, ML Engineer with 20 years of experience
Shreya Shankar,

Eliminate the guesswork of building AI applications with data-driven approaches for you and your team!

All students in this course get:

🗄️ Lifetime access to all materials!
♾️ Unlimited access to future cohorts: never worry about timing or missing out on new material.
🤖 6 months of unlimited access to our new AI Eval Assistant (more info below).
🧑‍🏫 10+ hours of office hours to maximize the value of live interaction.
🏫 A Discord community with continuous access to instructors to get unstuck (even after the course!).

Get Started Now!

🚨 The next cohort is January 2026. Enroll now and get immediate access to our course reader and community. 🚨

What you'll learn

Learn proven approaches for quickly improving AI applications. Build AI that works better than the competition, regardless of the use-case.

How To Collect Data For Evals

Understand instrumentation and observability for tracking system behavior.
Learn approaches for generating synthetic data to maximize error discovery and bootstrap product development.
Understand how to choose the right tools and vendors for you, with deep dives into the most popular solutions in the evals space.

Get Immediate Clarity And Direction With Error Analysis

Apply data analysis techniques to rapidly find systematic issues in your product regardless of the use case.
Master the processes and tools to annotate and analyze data quickly and efficiently.
Learn how to analyze agentic systems (tool calls, RAG, etc.) to quickly identify systematic patterns and errors.

Implement Effective Evaluations

Create evals that are customized to your product and provide immediate value, NOT generic off the shelf evals (which do not work).
Align evals with stakeholders & domain experts that allow you to scientifically trust the evals.
Create high-quality LLM-as-a-judge and code based evals with a systematic, iterative process.

Master Architecture-Specific Eval Strategies

Learn how to measure & debug RAG systems for retrieval relevance and factual accuracy.
Understand how to tame multi-step pipelines to identify error propagation and root-causes of errors quickly.
Master techniques that apply to multi-modal settings, including text, image, and audio interactions.

Run Evals In Production

Learn how to set up automated evaluation gates in CI/CD pipelines.
Understand methods for consistent comparison across experiments, including how to prepare and maintain datasets to prevent overfitting.
Implement safety and quality control guardrails.

Ensure That Evals Lead To High ROI

Develop a strong intuition of when to write an eval, and when NOT write an eval.
Learn how to design interfaces to remove friction from reviewing data and collect higher quality data with less effort.
Learn how to avoid common pitfalls surrounding team organization, collaboration, responsibilities, tools, automation, and metrics.

What's included

Live sessions

Learn directly from Hamel Husain & Shreya Shankar in a real-time, interactive format.

Lifetime Access to All Recordings & Materials

Revisit the materials and lectures anytime. Recordings and slides are made available to all students.

150+ Page Course Reader

We provide a course reader with detailed notes to supplement your learning and act as a future reference as you work on evals.

Lifetime Access To Discord Community

Private discord for questions, job leads, and ongoing support from the community (over 1000+ students and growing).

8+ Office Hour Q&As

Open office hours for questions and personalized feedback.

4 Homework Assignments With Solutions & Walkthroughs

Optional coding assignments & walkthrough videos so you can practice every concept.

Certificate of Completion

Share your new skills with your employer or on LinkedIn.

Detailed Vendor & Tools Workshops

Curated talks from industry experts working on evals, as well as workshops with vendors building eval tools.

For Groups and Teams

Opportunity for an exclusive Slack channel and private office hours.

Ranked the #1 Course on Maven.com

AI Evals For Engineers & PMs

We’ve worked with over 500 of the world’s top companies

Eliminate the guesswork of building AI applications with data-driven approaches for you and your team!

Get Started Now!

What you'll learn

Who this course is for:

What's included

Live sessions

Lifetime Access to All Recordings & Materials

150+ Page Course Reader

Lifetime Access To Discord Community

8+ Office Hour Q&As

4 Homework Assignments With Solutions & Walkthroughs

Certificate of Completion

Detailed Vendor & Tools Workshops

For Groups and Teams

Contact Us

AI Evals For Engineers & PMs

We’ve worked with over 500 of the world’s top companies

Eliminate the guesswork of building AI applications with data-driven approaches for you and your team!

Get Started Now!

What you'll learn

Who this course is for:

What's included

Live sessions

Lifetime Access to All Recordings & Materials

150+ Page Course Reader

Lifetime Access To Discord Community

8+ Office Hour Q&As

4 Homework Assignments With Solutions & Walkthroughs

Certificate of Completion

Detailed Vendor & Tools Workshops

For Groups and Teams

Contact Us

This website uses cookies.