Experimentation & A/B Testing: Prove It Before You Ship It
Signature Course No. 292 — A Signature AI Socratic Coaching Course.
No experience needed — your coach starts where you are.
A New Kind of Learning
Socratic coaching, delivered by an agentic AI coach.
Not a course you watch. An AI coach you work with.
The Lighthouse Coach begins by discovering how you think about the subject, teaches what is missing, and gives you realistic practice before asking you to prove that you can use it. You may test out of concepts you already understand, while capstones turn the learning into a practical plan, decision, conversation, or other real-world result.
The curriculum is deliberately authored; the AI delivers it through coaching. It is not an AI randomly generating lessons.
A coach that teaches by asking — not by lecturing
The Lighthouse Coach is an AI Socratic coach built on Cognosco Learning's Experimentation & A/B Testing: Prove It Before You Ship It course, in the Sales, Service & Revenue category. It does not play videos at you. It holds a conversation: it asks you a question, listens to your answer, names the misconception it hears, affirms what you got right, and asks the next question — the way a great coach works in the room.
Every team ships changes it believes in and then argues about whether they worked. This course replaces the argument with an instrument: a controlled experiment designed fairly, run cleanly, and read at exactly the strength its design supports. A result is only ever as strong as the design that produced it — that sentence is not a theme of the course; it is the subject of the course.
Module 1 builds the fair test. A hypothesis sharp enough to be wrong; randomization at the right unit; a baseline and its noise measured honestly; a sample size worked from the decision backwards; significance at its real strength; one deciding metric; guardrails that can stop a launch; sealed groups; and a protocol locked before launch.
It works in two movements. First, Module 1 — Design a Fair Test: Hypothesis, Randomization, Size, and Guardrails — work through Concepts 1–9, finishing with The Locked Protocol; then, Module 2 — Read It Without Fooling Yourself: Run Clean, Judge Honestly, Decide, and Report — work through Concepts 10–18, finishing with The Decision and the Honest Report.
Both capstones produce real artifacts — the locked test protocol: a complete, fair design for a checkout change, written before a single visitor sees it; and the decision memo and honest report: a finished test audited, read at its real strength, decided under its protocol's rule, and reported in the seven parts the next team can inherit.
The honesty standard is part of the course. Every statistical worked example recomputes — the arithmetic is shown, and the same figures recur verbatim. Every simplification is disclosed where it appears: the sample-size rule, the square-root behaviour of noise, the multiplicity working, the interval's working definition. And no result is read as stronger than its design supports — a randomized test licenses a causal claim about its population and period; a before-and-after comparison licenses less, and the course says so every time.
The course is self-paced and always on — roughly 30 minutes per concept, about 9 hours in full, taken in as many sittings as you like. You can type your answers, or talk out loud: the Voice Coach speaks its questions and listens to your spoken replies. Learners who already know the material can test out concept by concept.
Experimentation & A/B Testing: Prove It Before You Ship It is part of The Lighthouse Signature Series — the online counterpart to the one-day, in-person learning Cognosco Learning has delivered since 2002.
Course Facts
One full concept, coached end to end — free: verify your email and we’ll send you a link to start.
Already enrolled? Enter the course →
Diagnose → Learn & Practice → Prove → Capstone
Every learner moves through the same discipline a Cognosco instructor would insist on in the room.
Diagnose
The Coach starts where the course starts: with diagnosis. A fair test is built before it is run — a sharp hypothesis, a random split, an honest baseline, enough observations to answer, one deciding metric, guardrails against harm, sealed groups, and a protocol locked before launch. Every concept opens with three questions that locate where you actually stand, and your answers shape how deeply the Coach works the concept with you. Everyone begins with Module 1 — no skipping the foundations you only think you have.
Learn & Practice
Work each concept in conversation with the Coach — spoken or written — then practice it against realistic situations until it holds up.
Prove
Every concept ends in proof: answer the check cold and demonstrate the skill. Know it already? Test out and take credit immediately.
Capstone
Finish with two capstones — The Locked Protocol for Module 1 and The Decision and the Honest Report for Module 2 — that pull the whole course together, and earn the certificate.
What You'll Learn
Design before data. Almost every dishonest result was decided before launch — by a vague hypothesis, a self-selected group, a metric chosen after the numbers arrived, or a sample too small to answer anything. Module 1 is spent entirely on design because design is where honesty is won or lost; reading well cannot rescue a test that was never fair.
Module 1 builds the instrument. The claim made sharp enough to fail; the split made by chance at the right unit; the ground measured with its ordinary range; the question sized so the traffic can answer it; the judging instruments — threshold, deciding metric, guardrails — built before they are needed; the groups sealed; and the whole design locked in writing.
Module 2 protects the instrument from its operators. Mid-flight checks that catch a broken split on day one; the peeking, slicing, and calendar traps refused; the interval judged at its real strength; honest work on weaker ground when randomization is impossible; and the payoff sequence — the decision under the protocol's rule, the phased rollout behind a rollback line, and the report written at exactly the strength the design earned.
And it holds its honesty throughout. The running case — a checkout test and a search test at a fictional online retailer — carries stipulated teaching figures, stated once and used consistently, so you can check the arithmetic wherever it recurs. Simplifications are disclosed at the point of use, never hidden.
What you will learn
- Design a fair test: a testable hypothesis, clean randomization, an honest baseline, and a sample size that can answer the question
- Judge evidence at its real strength: significance, intervals, practical importance, and the traps of peeking, multiplicity, and time effects
- Protect people and the business while you test: one deciding metric, guardrails with thresholds, uncontaminated groups, and a locked protocol
- Decide and report honestly: ship, iterate, or kill; phased rollout with monitoring; and a report that states what was proven and what was not
Everything in the course
- All 18 concepts of Experimentation & A/B Testing: Prove It Before You Ship It, in 2 modules
- A starting-point check first — the Coach finds where you actually stand before teaching
- Learn, Practice, Prove for every concept — no passive video watching
- A test-out on every concept: prove you know it and move on
- The Voice Coach: talk with the Coach out loud — it speaks, listens, and replies
- A personal Learner Record of your work across the course
- Two capstone exercises: The Locked Protocol and The Decision and the Honest Report
- A printable session summary whenever you pause
- The Lighthouse Coach completion certificate (Signature Series)
- Your progress is saved as you go — stop anytime, resume where you left off
Built for the people who are asked “did it work?”
Growth, product, and marketing analysts — who design tests, read results, and answer for them in rooms that want a yes.
Product managers, designers, engineers, and data staff — who propose changes, or build the assignment, logging, and exposure checks the discipline depends on.
Managers who approve launches — and anyone who has shipped a “winner” that later did nothing. The course assumes ordinary business arithmetic, every calculation shown with its working, and no prior statistics beyond what the book itself teaches.
Prefer to learn live and in person? The same material anchors Cognosco's one-day programs — browse the in-person catalog.
Start with Design a Fair Test
$195 — introductory price. One payment. No subscription, no password to manage.
Here is exactly what happens when you enroll:
- Pay securely with Stripe — you are taken straight into the course
- Create your learner profile and begin the Module 1 starting-point check
- Save the access key you are given — it restores your enrollment on any device
- Learn at your pace; your progress is stored in your browser on your device
For the full voice experience, use Chrome and allow the microphone when asked. Typing your answers works everywhere, all the time. Questions before you enroll? Email hello@cognoscolearning.com.
Before you enroll
What is a Socratic coaching session?
A conversation, not a video. Your coach asks a question; you answer — out loud by voice, or by typing — and the coach responds to what you said: naming the misconception it hears, affirming what you got right, and asking the next question. You advance only when you have mastered each concept, so a finished course means the learning held.
How long does it take — and how long do I have?
About 30 minutes per concept. Learn in sittings of any length; your progress is saved as you go. You have a full year from purchase to finish, and you can revisit and retake within that year.
What does the certificate verify?
That you completed every concept and both capstones of this course. Anyone can confirm a Cognosco certificate online — enter its certificate ID on our verification page.
What if it isn't for me — what are the refunds?
If you have not started, cancel for a full refund within seven days of purchase. Once you have started, there is no cash refund — stop before finishing and we credit the full amount you paid toward any other course in the library (credit valid one year from your original purchase). Once you complete the course, no refund or credit applies.
More Signature Courses are coming
Experimentation & A/B Testing: Prove It Before You Ship It is Signature Course No. 292. Leave your email and we'll send you news of future Signature Courses and Lighthouse updates as they are announced — nothing else.