Training Session · Analytics & Decision Making

Product Analytics &

Experimentation

Data fluency isn't about knowing SQL. It's about knowing which questions to ask, which metrics to trust, and how to design experiments that actually tell you something. This workshop builds that muscle.

FormatWorkshop
AudienceProduct Managers, Analysts & Product Leaders
Agenda4 Blocks + Exercises
Status↗ Ongoing
Session Overview

What This Workshop Covers

From metric design to experiment analysis — the analytical foundations that separate good product decisions from lucky ones.

◈
Metrics Architecture
How to build a metrics system that connects user behavior to business outcomes — and how to spot metrics that look good but mislead.
⇄
Experiment Design
A/B testing from first principles — sample size, statistical power, and the decisions that determine whether your experiment results are trustworthy.
∿
Causal Inference
Beyond correlation — when you can't run an A/B test, how to get closer to causality with observational data and natural experiments.
⬡
Communicating Results
Turning analysis into decisions — how to present experiment results to stakeholders who will push back, simplify, or misinterpret.
Session Structure

The Agenda

Four blocks that take you from metric design through experiment analysis to communicating results under uncertainty.

Block 01

Designing Metrics That Matter

The metrics hierarchy: input metrics, output metrics, guardrail metrics — and why you need all three
Vanity metrics vs actionable metrics — the subtle difference and how to tell them apart in practice
Metric trees: mapping from business outcomes down to the user behaviors that drive them
Exercise Build a metric tree for a product feature — start from a business goal and map every user behavior that would move it
Block 02

A/B Testing from First Principles

Statistical power, sample size, and significance — what these numbers actually mean and why most PMs get them wrong
The multiple comparison problem — why testing 10 variants at once quietly inflates your false positive rate
Pre-registration and test hygiene — how to design an experiment before you look at results
Key Distinction Statistical significance tells you the result probably isn't noise. It doesn't tell you the result matters.
Block 03

When You Can't Run an Experiment

Observational studies and their limits — what you can infer from data when randomization isn't possible
Difference-in-differences and natural experiments — finding causal signals in real-world product data
Regression discontinuity and synthetic controls — advanced techniques for the hardest causal questions
Exercise Design an observational study for a product change that can't be A/B tested — define the comparison group, identify confounders, and state the assumptions your inference rests on
Block 04

Turning Analysis into Decisions

Interpreting results under uncertainty — how to communicate confidence intervals without losing the room
When to ship despite ambiguous results — the decision framework for acting on imperfect data
Common ways analysis gets misused in product reviews — and how to defend against it
What You Leave With

Three Tools for Better Decisions

Session Frameworks

Apply these the day after the workshop

Practical frameworks designed to change how your team designs metrics, runs experiments, and communicates results to stakeholders.

◈
The Metrics Canvas
A structured template for building a metric tree — from business outcome to user behavior to measurable event, with guardrail metrics identified at each level.
⇄
Experiment Design Checklist
A pre-flight checklist for A/B tests — covering sample size, test duration, success criteria, and the hygiene requirements that make results trustworthy.
⬡
Results Communication Guide
A framework for presenting experiment results to stakeholders — structured to handle pushback, surface uncertainty honestly, and still drive a clear recommendation.
Skills Covered

What You'll Be Able to Do

Product Analytics Experimentation Design Causal Inference Metrics Architecture A/B Testing Statistical Significance Sample Size Calculation Metric Trees Observational Studies Difference-in-Differences Data Storytelling Guardrail Metrics

Bring this workshop to your product or data team.

Available as a half-day or full-day session. Designed for PMs and analysts who want to go beyond dashboards. Get in touch.

Get in Touch ← All Courses