Correlation, Significance & Traps — Causation, p-values & A/B Testing

Q: Why is correlation not causation?

Two variables can move together because of coincidence, reverse causation or a hidden confounding variable — so correlation alone can't prove one causes the other.

Q: What does a p-value mean?

The probability of seeing a result at least this extreme if there were no real effect. A small p-value (e.g. < 0.05) suggests the result is unlikely to be chance — but significance isn't the same as importance.

Q: What is an A/B test?

A controlled experiment that randomly splits users into a control and a variant, changes one thing, and compares a metric — randomisation lets you attribute the difference to the change.

Correlation, Significance & Avoiding Traps: Causation, p-values and A/B Testing

2 min readPublished 29 Jul 2026

This is where analysts earn their credibility: turning patterns into sound, defensible conclusions without falling into the classic traps. You don't need advanced maths — you need clear reasoning.

Correlation vs causation

Correlation measures how two variables move together, from -1 (perfect inverse) through 0 (no linear relationship) to +1 (perfect positive). It's genuinely useful — but correlation does not prove causation. Ice-cream sales and drownings correlate (both rise in summer), but one doesn't cause the other — a hidden confounding variable (hot weather) drives both. Before claiming X causes Y, ask: could a third factor explain it? Could it be reverse causation? Could it be coincidence?

Statistical significance and p-values (plainly)

When you see a difference in data, is it real or just random noise? A significance test helps answer that. The p-value is the probability of seeing a result at least this extreme if there were actually no real effect. A small p-value (commonly < 0.05) suggests the result is unlikely to be chance, so it's "statistically significant". Key cautions:

Significance is not importance — a tiny, meaningless effect can be "significant" with enough data.
A non-significant result doesn't prove "no effect" — you may just lack data.
Beware p-hacking — testing many things until something looks significant.

Sample size matters

Small samples are unreliable and swing wildly; larger, representative samples give more trustworthy results. A pattern from 20 customers is a hunch; from 20,000 (if representative) it's evidence. Always check how much data a claim rests on — and whether it's biased.

A/B testing basics

The cleanest way to establish cause is a controlled experiment: split users randomly into A (control) and B (variant), change one thing, and compare a chosen metric. Randomisation removes confounders, so a significant difference can be attributed to the change. Watch for adequate sample size and running the test long enough.

Put it to work

Next time someone says "X caused Y", check for confounders, sample size and whether the difference is meaningful — not just "significant". Practise evidence-based recommendations in the Sales Analysis lab.

Interview Intelligence

How this topic actually shows up in interviews — and how to demonstrate you understand it.

Why employers ask about this

Employers trust analysts who reason soundly about evidence and don't over-claim from data.

Technical questions

What's the difference between correlation and causation?+

Correlation is co-movement; causation means one drives the other. Check for confounders, reverse causation and coincidence before claiming cause.

What does statistical significance tell you — and not tell you?+

That a result is unlikely due to chance; it does NOT tell you the effect is large or important, or prove there's no effect if non-significant.

Behavioural questions

Tell me about a time you avoided drawing a wrong conclusion from data.+

Use STAR: how you spotted a confounder, small sample or a significant-but-trivial effect, and gave an honest, caveated recommendation.

Real-world scenarios

“Marketing says a new banner 'caused' a sales rise because both went up.”+

Expected answer: Check for confounders and seasonality, look at sample size, and recommend a proper A/B test to establish cause before crediting the banner.

Practise in AI Interview™Rehearse these questions, scored.Plan with Career Coach™A personalised path to your goal.Evidence it in Career Passport™Prove this skill to employers.

Employability Intelligence

Where this knowledge takes you — the jobs, skills and certifications it feeds into.

Relevant roles

Insights AnalystData AnalystBI Analyst

Skills you're proving

Correlation vs causationSignificance / p-valuesSample sizeA/B testing

Recommended certifications

Microsoft Power BI (PL-300)Google Data Analytics

Career progression

Distinguishes credible analysts and unlocks insights/senior roles.

What employers expect

That you can clean, analyse and visualise real data, write SQL, and turn findings into clear, actionable recommendations for the business.

Frequently asked questions

Why is correlation not causation?

Two variables can move together because of coincidence, reverse causation or a hidden confounding variable — so correlation alone can't prove one causes the other.

What does a p-value mean?

The probability of seeing a result at least this extreme if there were no real effect. A small p-value (e.g. < 0.05) suggests the result is unlikely to be chance — but significance isn't the same as importance.

What is an A/B test?

A controlled experiment that randomly splits users into a control and a variant, changes one thing, and compares a metric — randomisation lets you attribute the difference to the change.

Practise what you've learned

Turn this guide into real, evidenced progress

Missiora helps you measure, improve and evidence the capabilities employers actually value — start with the tools best suited to this topic.

Published by

Missiora

Missiora is an AI Employability Intelligence platform. Our resources are researched and reviewed by the Missiora team to help you measure, improve and prove your career readiness.