Chapter 01

Why PMs Need This

The data and AI foundations every PM needs — without becoming a data scientist.

⏱ 10 min read 🔰 Start here

Everyone has a different interpretation of the same number

Picture this: your data team shares a slide. "Engagement is up 12% after the redesign." The room nods. Someone says "great work." You move on.

But nobody asked: 12% compared to what? Over what time window? Which users? Was anything else launched that week? Is 12% even meaningful given how noisy this metric is — meaning how much it bounces around randomly even when nothing real has changed?

This happens in almost every product review, every week. And the cost isn't just making a bad call on one feature — it's building a habit of mistaking noise for signal, correlation for causation, and activity for progress.

PM Insight

You don't need to run the analysis. You need to know which questions to ask before trusting it. That's the skill this guide builds.

The same dynamic plays out when you're building AI features. Your team ships a personalisation model. It performs well in testing. Six weeks in, recommendations start missing. Your DS lead says the input feature distribution shifted. Your job isn't to fix the model — it's to know what questions to ask, what decisions are yours, and what tradeoffs you're being asked to make. Right now, that conversation probably feels opaque. By Chapter 9, it won't.

Three roles data plays in product work

Data shows up differently depending on what stage of a decision you're in.

1. Data as evidence

You're deciding whether a problem is real and how big it is. Here, data answers: how many users hit this? how often? how much does it cost them? The risk is cherry-picking — finding data that confirms what you already believe.

2. Data as signal

You've shipped something and want to know if it worked. Here, data answers: did behavior change? did it change because of what we did? The risk is attribution — many things change at once, and it's easy to credit the thing you shipped.

3. Data as input to ML

You're building a feature that learns from data — a recommendation engine, a ranker (a system that orders results, like a search page or a feed), or a classifier (a system that puts things into categories, like a spam filter or a fraud detector). Here, data answers: what patterns exist that a model can learn? The risk is garbage in, garbage out — and not knowing that's what happened.

PM Insight

Each role requires different skepticism. "Is this problem real?" needs different rigor than "did this launch work?" — and both are different from "is this model good enough to ship?" Knowing which question you're answering changes what you should scrutinize.

What PMs actually need to know

The job isn't to run the analysis — it's to know which questions to ask, and what to do with the answers. That requires a specific mental toolkit:

1 — Distributions, not just averages

When someone reports an average, your first instinct should be: average of what? What does the spread look like? A feature used "on average 3 times a week" might be used daily by 10% of users and never by the rest.

2 — Correlation vs causation

Two metrics moving together doesn't mean one caused the other. Both might be driven by a third thing you're not measuring. This is the most common error in product analytics.

3 — Statistical significance vs practical significance

A result can be statistically significant (very unlikely to be random) and practically meaningless (too small to matter). With large enough samples, almost anything clears the significance bar.

4 — Model confidence ≠ model accuracy

"87% confident" is the model's internal probability score — how strongly it's leaning toward a prediction. It is not "this prediction is correct 87% of the time." What matters is the actual error rate: of every 100 users flagged as churners, how many actually churn? That's what tells you whether to act.

5 — What's feasible vs what sounds feasible

"Can't we just use AI to predict X?" is asked in every product meeting. The answer depends on whether you have labelled data (historical examples where you already know the outcome — e.g. past users who churned vs stayed), whether a predictive signal even exists, and whether the business case covers the engineering cost. This guide helps you ask those questions.

What this course builds toward

This isn't a data analytics course. It's a course about how to work effectively at the intersection of product and AI — starting with the foundations that make everything else legible.

Chapters 1–5 — Foundations

Stats, probability, A/B testing, and metrics. The concepts your data team uses every day — and the ones you need to ask the right questions and spot bad analysis before it shapes a bad decision.

Chapters 6–10 — How ML works

Under the hood of machine learning: how models learn, how they fail, how they're evaluated. Not to build them — to know what you're shipping, what can go wrong, and what "good enough" actually means.

Chapters 11–14 — AI & data in practice

LLMs, AI product decisions, working with data teams, and data infrastructure. The decisions that land on your desk when AI is part of the product — and how to make them with confidence.

What you'll be able to do

How to use this course

The early chapters build vocabulary the later ones use, so reading in order pays off. Wherever a concept is clearer to see than to read, there's an interactive visualisation — the intuition from dragging a slider is stickier than a definition. Every chapter ends with a PM Playbook: the specific questions to ask in real meetings, scoped to that chapter's topic.

PM Playbook — Questions to ask right now

The next time someone shares analysis in a meeting, try these:

These six questions alone will change the quality of decisions your team makes.

Check your understanding 4 questions