P-value

One-Line Definition

A P-value is the probability of observing a result at least as extreme as the one you actually got, assuming the null hypothesis is true — it tells you how surprising your data would be if there were really no effect.

That's the whole idea. Everything else on this page is about using it correctly, which is where most people (and most A/B test dashboards) go wrong.

Real-Life Analogy: The Suspicious Coin

Imagine a friend hands you a coin and claims it's fair. You flip it 100 times and get 63 heads.

If the coin were truly fair, how unusual would 63 heads be? You can work this out. Under a fair coin (p = 0.5), the standard deviation of heads in 100 flips is √(100 × 0.5 × 0.5) = 5. So 63 heads is (63 − 50) / 5 = 2.6 standard deviations above the expected 50. The two-tailed probability of landing that far out is roughly 0.009, or 0.9%.

That 0.009 is your P-value. It doesn't tell you the coin *is* rigged. It tells you that *if* the coin were fair, a result this lopsided would happen fewer than 1 time in 100 experiments. You now have to decide whether "the coin is fair" or "this was a 1-in-100 fluke" is the more believable story.

That decision — not the P-value itself — is the actual judgment call.

Core Formula

For a test statistic $T$ computed from your sample, and a null hypothesis $H_0$:

$$

p = P(T_{\text{extreme}} \geq T_{\text{observed}} \mid H_0)

$$

In words: hold $H_0$ fixed as if it were true, then ask how much of the probability mass sits at or beyond your observed result.

A concrete worked example for a two-proportion z-test (the workhorse of e-commerce A/B testing):

$$

z = \frac{\hat{p}_B - \hat{p}_A}{\sqrt{\bar{p}(1-\bar{p})\left(\frac{1}{n_A} + \frac{1}{n_B}\right)}}

$$

Suppose your control converts at 4.0% (n = 12,000) and your variant at 4.5% (n = 12,000). Pooled rate $\bar{p}$ = 0.0425.

- Numerator: 0.045 − 0.040 = 0.005

- Denominator: √(0.0425 × 0.9575 × (1/12000 + 1/12000)) ≈ 0.00260

- $z$ ≈ 1.92

Two-tailed P-value ≈ 0.055. That's above the conventional 0.05 threshold — so the "standard" answer is "not significant," even though the variant genuinely looks better. This is exactly the kind of edge case that causes arguments in growth meetings.

Comparison with Related Terms

The critical row is the last one. A P-value of 0.03 does not mean there's a 97% chance your variant works. It means there's a 3% chance of seeing data this extreme *if the variant does nothing*.

Use Cases in DTC & Cross-Border E-commerce

1. A/B testing landing pages. You test a new hero image across 40,000 sessions split evenly. Conversion lifts from 2.8% to 3.1%. The P-value comes out at 0.04 — significant at α = 0.05. But the absolute lift is 0.3 percentage points. If your AOV is $45, that's roughly $0.135 per session. Whether that clears your cost of implementation is a business question the P-value cannot answer.

2. Creative fatigue detection. You run the same Meta ad for 6 weeks. Week 1 CTR is 1.9%; week 6 CTR is 1.2%. A P-value on the difference tells you whether the drop is real decay or normal variance. With small daily samples, you'll often get p = 0.20 and wrongly conclude "no fatigue" when you simply lack power.

3. Supplier / 3PL quality checks. You audit 500 outbound parcels from a new fulfillment partner and find 22 with damaged packaging (4.4%) versus your incumbent's historical 3.0%. P-value ≈ 0.07. Not significant at 0.05 — but on a 100,000-order-per-month volume, that gap is 1,400 extra damaged parcels. Significance and materiality diverge sharply here.

4. Multi-market rollout decisions. Testing a localized checkout in three markets simultaneously (DE, FR, UK) and running three separate P-values invites false positives. With α = 0.05 across three independent tests, your family-wise error rate is 1 − 0.95³ ≈ 14%. Bonferroni correction (α = 0.0167 per test) or a Bayesian hierarchical model is the fix.

5. Email subject line tests. Small samples, high variance, and open rates that swing on send-time. P-values here are notoriously unstable — a test that reads p = 0.02 on Tuesday can read p = 0.31 on Thursday with 20% more data.

Common Misconceptions

"p = 0.04 means there's a 4% chance the null is true." No. It's P(data | null), not P(null | data). Confusing these is the base rate fallacy, and it's the single most common error in e-commerce analytics.

"p > 0.05 means no effect." Absence of evidence is not evidence of absence. A test with 2,000 sessions per arm has very low power to detect a 0.2 pp lift. You'll get p = 0.60 and learn almost nothing.

"A smaller P-value means a bigger effect." P-values shrink with sample size even when the effect size stays constant. Run a trivial 0.05 pp lift on 5 million sessions and you'll get p < 0.001. The effect is real and utterly irrelevant.

"p = 0.049 is meaningful, p = 0.051 is not." This is threshold worship. The two results carry nearly identical information. Treat P-values as a continuous measure of evidence, not a binary gate.

"P-hacking is fine if I don't get caught." Peeking at your test daily and stopping when p < 0.05 inflates your false positive rate to 20–30% easily. Pre-register your sample size, or use sequential testing methods designed for it.

"Statistical significance = ship it." Significance says nothing about implementation cost, brand risk, or whether the lift persists past week two. It's one input among many.

Related Terms

- Null hypothesis (H₀) — the default assumption of no effect that the P-value is computed against

- Alternative hypothesis (H₁) — the effect you're trying to detect

- Alpha (α) — your pre-declared significance threshold, conventionally 0.05

- Type I error — false positive; rejecting a true null

- Type II error — false negative; failing to reject a false null

- Statistical power — 1 − β; probability of detecting a real effect

- Effect size — magnitude of the difference, independent of sample size

- Confidence interval — range estimate that conveys both effect size and precision

- Multiple comparisons correction — Bonferroni, Benjamini-Hochberg, and friends

- Bayesian inference — an alternative framework that gives P(effect | data) directly

- Minimum detectable effect (MDE) — the smallest lift your test can reliably catch

Bottom line: A P-value is a measure of surprise, not a measure of truth, size, or importance. Use it alongside effect sizes, confidence intervals, and business context — never on its own.