Retour au Catalogue
AI engineering

Evals for AI Agents: From Vibes to Evidence

Every change to a prompt, a tool or a model changes your agent's behaviour, and « I tried it a few times and it seemed fine » is how regressions reach users. Evals turn that feeling into a number you can trust. You'll build a golden set from real cases, choose graders that check outcomes instead of wording, measure pass rates instead of single runs, and wire it all into CI as a gate. You leave with a working harness you can point at your own agent.

6 h
E-learning
Gratuit

Sessions organised on request

Evals for AI Agents: From Vibes to Evidence
Gratuit
Paiement sécurisé

Objectifs pédagogiques

  • Create a versioned golden set with expected outcomes or rubrics
  • Implement at least two grader types and calibrate a model grader against human labels
  • Run an agent eval harness with claude -p and structured output
  • Add an eval regression gate to CI

Public visé

Developers and ML/AI engineers who ship prompts or agents to real users, and tech leads who decide when a change is good enough.

Prérequis

Basic JavaScript or TypeScript (you write small node:test checks and extend a Node script; the kit is Node-only); macOS, Linux or Windows with WSL 2; git, and Docker for running the harness in a container. Node.js 22.6+, Claude Code 2.1.295+ and an Anthropic API key with a low spend limit (a Console account with credits): the harness exercises (2, 2b, 3, 4) run in a container where you must not log in, so a subscription login alone isn't enough. It is enough for the model grader in exercise 2b on your machine. Each agent run is capped at $1 and cost about $0.25 in our tests: exercise 2 is 15 runs, each CI PR 10 runs, the final project 30 cases × 3 runs (use --k 1 while you iterate).

Programme de formation

  1. Why evals · 2. A golden set · 3. Graders · 4. Evaluating agents · 5. Evals as a CI gate · 6. Final project

Moyens pédagogiques et techniques

Short explanations with diagrams, a working eval harness and five graded coding tasks in the kit, four hands-on exercises and a final project on your own agent.

Modalités d'évaluation

  • Short checkpoint quiz on every page (one mistake allowed), final exam at 70 %
  • Three hands-on exercises with a rubric
  • Final project: an eval suite with a CI gate, reference solution provided

Durée et rythme

Self-paced: about 1 h 15 of lessons, plus 4–6 h for the exercises and the final project.

Conditions d'accès

Free. Create a learner account on the course page and start immediately.

Délais d'accès

Immediate access after enrolment.

Accessibilité aux personnes en situation de handicap

Self-paced, no time limit. Diagrams have text descriptions, all code is text.

Tarif

Gratuit

Funding

No funding needed.

Recognition

Non-certifying course: you receive a certificate of completion.

Dispensé par

ITwithDomITwithDom

Contact

Dominik Heinrich (ITwithDom)