AI Evaluation in Practice: Test, Score and Red-Team the Models You Ship
Signature Course No. 433 — A Signature AI Socratic Coaching Course.
No experience needed — your coach starts where you are.
A New Kind of Learning
Socratic coaching, delivered by an agentic AI coach.
Not a course you watch. An AI coach you work with.
The Lighthouse Coach begins by discovering how you think about the subject, teaches what is missing, and gives you realistic practice before asking you to prove that you can use it. You may test out of concepts you already understand, while capstones turn the learning into a practical plan, decision, conversation, or other real-world result.
The curriculum is deliberately authored; the AI delivers it through coaching. It is not an AI randomly generating lessons.
A coach that teaches by asking — not by lecturing
The Lighthouse Coach is an AI Socratic coach built on Cognosco Learning's AI Evaluation in Practice: Test, Score and Red-Team the Models You Ship course, in the AI at Work category. It does not play videos at you. It holds a conversation: it asks you a question, listens to your answer, names the misconception it hears, affirms what you got right, and asks the next question — the way a great coach works in the room.
A passing evaluation never proves a model safe or correct. An evaluation measures what it measures — these inputs, this scoring method, this day. It says nothing by itself about the inputs you did not test, the failures your scoring method cannot see, or the model your vendor quietly swaps in next quarter. This course teaches you to judge the models you ship — and to say, precisely, how much a score is allowed to mean.
It starts with defining the job. Every failed evaluation traces back to the same skipped step: nobody wrote down, precisely, what the model is supposed to do. You will write the five-part job definition — user and moment, real input shape, output contract, failure costs ranked, and the boundary of the claim — then build the golden set and the rubric that turn ‘good answer’ from an opinion into an instrument.
It works in two movements. First, Module 1 — Judge It Before It Ships — work through Concepts 1–9, finishing with The Evaluation Plan; then, Module 2 — Keep It True After It Ships — work through Concepts 10–18, finishing with The Ship-Decision Report.
Both capstones produce real artifacts — a complete evaluation plan for Qorvane Systems' new Returns Triage Assistant: the job definition, golden-set design, rubric, scoring design, metrics and slices, and a ship bar written six weeks before any score exists; and a ship-decision report on a changed model for the Support Copilot: regression posture, probing results, live-watch evidence, a paired comparison against the standing bar, and a signed recommendation with its limits printed.
The honesty standard is part of the course. The bar is set before the number is seen — golden sets frozen before tuning begins, ship bars written before scores arrive, comparison tests designed before either model runs. Evaluation done in the other order is not evaluation; it is advocacy with arithmetic. And the running company is fictional, its figures stipulated and consistent, so you can recompute every worked example yourself.
The course is self-paced and always on — roughly 30 minutes per concept, about 9 hours in full, taken in as many sittings as you like. You can type your answers, or talk out loud: the Voice Coach speaks its questions and listens to your spoken replies. Learners who already know the material can test out concept by concept.
AI Evaluation in Practice: Test, Score and Red-Team the Models You Ship is part of The Lighthouse Signature Series — the online counterpart to the one-day, in-person learning Cognosco Learning delivers.
Course Facts
One full concept, coached end to end — free: verify your email and we’ll send you a link to start.
Already enrolled? Enter the course →
Diagnose → Learn & Practice → Prove → Capstone
Every learner moves through the same discipline a Cognosco instructor would insist on in the room.
Diagnose
The Coach starts where the course starts: with the job definition. Take the AI feature closest to your own work and state its job in one sentence, as your team usually states it — then list what that sentence leaves undecided; recall how your team last settled a disagreement about whether a model answer was good, and whether a written standard or seniority settled it; and name what your feature currently gets credit for that nobody has actually tested. Every concept opens with three questions that locate where you actually stand, and your answers shape how deeply the Coach works the concept with you. Everyone begins with Module 1 — no skipping the foundations you only think you have.
Learn & Practice
Work each concept in conversation with the Coach — spoken or written — then practice it against realistic situations until it holds up.
Prove
Every concept ends in proof: answer the check cold and demonstrate the skill. Know it already? Test out and take credit immediately.
Capstone
Finish with two capstones — The Evaluation Plan for Module 1 and The Ship-Decision Report for Module 2 — that pull the whole course together, and earn the certificate.
What You'll Learn
Define it, test it, score it, slice it — and set the bar before you see the number. Every skill in this course is the same discipline at a different depth: say exactly what the model is for, measure it against cases drawn from reality, score it with an instrument two honest people apply the same way, and read the result in slices until the failures hiding inside the average show themselves.
Module 1 builds the pre-launch practice. Define the Job Before You Measure It; Build a Golden Set That Earns Trust; Write Rubrics Two Raters Apply the Same Way; Score With People Without Fooling Yourself; Let a Model Grade a Model; Pick Metrics That Match the Cost of Being Wrong; Slice the Score Until the Failures Show; Set the Ship Bar Before You See the Number; and Read a Public Benchmark Without Being Sold.
Module 2 keeps the practice alive after launch. Turn Every Failure Into a Regression Test; Red-Team as Test Design, Not Theater; Test the Inputs Users Actually Send; Watch Quality After Launch, On Purpose; Tell Drift From Noise Before Users Tell You; Price the Eval Like It Costs Something; Investigate a Quality Drop Without Guessing; Decide Whether the New Model Ships; and Write the Eval Report That Tells the Truth.
And it holds its fences throughout. Securing the AI a company uses is taught in its own neighboring course, and deploying models in another. This course owns judging: how you define what good means for an AI feature, how you build the test that measures it, how you score the answers without deceiving yourself, and how you decide — with evidence you can defend — whether a model is good enough to put in front of people, and whether it still is next month.
What you will learn
- Define an AI feature's job precisely enough to test it, and build golden sets and rubrics that deserve trust
- Score model answers with human reviewers and with model graders — and name exactly where each method deceives you
- Choose metrics that match the real cost of each kind of error, slice results to find hidden failures, and set ship bars you hold under pressure
- Turn production failures into regression tests and probe a model adversarially as a discipline of test design
- Monitor live quality, distinguish drift from noise, and investigate a quality drop to its actual cause
- Budget evaluation effort where it buys the most certainty, decide model changes on paired evidence, and report results honestly — limits included
Everything in the course
- All 18 concepts of AI Evaluation in Practice: Test, Score and Red-Team the Models You Ship, in 2 modules
- A starting-point check first — the Coach finds where you actually stand before teaching
- Learn, Practice, Prove for every concept — no passive video watching
- A test-out on every concept: prove you know it and move on
- The Voice Coach: talk with the Coach out loud — it speaks, listens, and replies
- A personal Learner Record of your work across the course
- Two capstone exercises: The Evaluation Plan and The Ship-Decision Report
- A printable session summary whenever you pause
- The Lighthouse Coach completion certificate (Signature Series)
- Your progress is saved as you go — stop anytime, resume where you left off
Built for the people who own the ship decision
AI product managers and product engineers — who own the ship decision for a model or AI feature and need evidence that survives a hard question from a customer, an executive, or a regulator; and machine learning and platform engineers who build the prompts, retrieval, and model integrations, and who need a test practice that catches regressions before users do.
Quality, support, and operations leads — whose teams see the failures first, and who need a disciplined way to turn what they see into tests, scores, and fixes.
Founders and technical leaders at smaller companies — who cannot afford a dedicated evaluation team and need a practice sized to reality: rigorous where the risk is, cheap where it is not. This course assumes you work near an AI feature — you can describe what it does, who uses it, and what a bad answer costs. It assumes no statistics degree and no prior evaluation program.
Prefer to learn live and in person? The same material anchors Cognosco's one-day programs — browse the in-person catalog.
Start with Define the Job Before You Measure It
$195 — introductory price. One payment. No subscription, no password to manage.
Here is exactly what happens when you enroll:
- Pay securely with Stripe — you are taken straight into the course
- Create your learner profile and begin the Module 1 starting-point check
- Save the access key you are given — it restores your enrollment on any device
- Learn at your pace; your progress is stored in your browser on your device
For the full voice experience, use Chrome and allow the microphone when asked. Typing your answers works everywhere, all the time. Questions before you enroll? Email hello@cognoscolearning.com.
Before you enroll
What is a Socratic coaching session?
A conversation, not a video. Your coach asks a question; you answer — out loud by voice, or by typing — and the coach responds to what you said: naming the misconception it hears, affirming what you got right, and asking the next question. You advance only when you have mastered each concept, so a finished course means the learning held.
How long does it take — and how long do I have?
About 30 minutes per concept. Learn in sittings of any length; your progress is saved as you go. You have a full year from purchase to finish, and you can revisit and retake within that year.
What does the certificate verify?
That you completed every concept and both capstones of this course. Anyone can confirm a Cognosco certificate online — enter its certificate ID on our verification page.
What if it isn't for me — what are the refunds?
If you have not started, cancel for a full refund within seven days of purchase. Once you have started, there is no cash refund — stop before finishing and we credit the full amount you paid toward any other course in the library (credit valid one year from your original purchase). Once you complete the course, no refund or credit applies.
More Signature Courses are coming
AI Evaluation in Practice: Test, Score and Red-Team the Models You Ship is Signature Course No. 433. Leave your email and we'll send you news of future Signature Courses and Lighthouse updates as they are announced — nothing else.