Correlation, dependence, and the gap between them

What correlation actually computes

The Pearson correlation coefficient is the covariance of two variables divided by the product of their standard deviations:

r = cov(X, Y) / (sd(X) * sd(Y))

It runs from −1 to +1. It answers exactly one question: how well does a straight line describe the relationship? Not whether there is a relationship. Whether a straight line captures it.

Zero correlation, perfect relationship

Take y = x² with x spread symmetrically around zero: x = −3, −2, −1, 0, 1, 2, 3, so y = 9, 4, 1, 0, 1, 4, 9.

The correlation is exactly 0. Knowing x tells you y precisely — there is no uncertainty at all — and Pearson reports no association, because the best-fitting straight line through those points is flat. The upward half and the downward half cancel.

This is not a contrived exception. U-shaped relationships are everywhere: performance against arousal, engagement against notification frequency, quality against temperature in a language model. A feature-selection step that ranks features by correlation with the target will drop every one of them.

Strong correlation, no relationship

The reverse failure is more famous and more dangerous. Correlations appear between quantities with no mechanism connecting them, and they appear more often than intuition suggests because people look at many pairs.

Three mechanisms produce them:

• A common cause. Ice-cream sales and drowning deaths correlate, both driven by hot weather. Neither causes the other.
• Selection. Among people admitted to a competitive university, test scores and interview scores may correlate negatively, because admission required at least one to be high. The correlation is created by the filtering, not by the world. This is called collider bias, and it appears whenever you analyse a sample that was selected on an outcome.
• Chance across many comparisons. Test 100 pairs of unrelated variables at the 5 per cent level and about 5 will look significant. This is the arithmetic from the counting lesson, and it is why exploratory correlation matrices produce so many findings that do not replicate.

Independence is stronger than uncorrelated

Two variables are independent when knowing one changes nothing about the distribution of the other. Uncorrelated means only that the linear component of the association is zero. Independence implies zero correlation; zero correlation does not imply independence, as the y = x² example shows.

The one place they coincide is jointly normal variables, where zero correlation does mean independence. That special case is why the two ideas get conflated: a great deal of classical statistics assumes normality, and under normality the distinction disappears. Your data is generally not jointly normal.

Measures that catch what correlation misses

• Spearman correlation is Pearson applied to the ranks. It catches any monotone relationship, including curved ones, and it is robust to outliers. If you compute one correlation, compute this one alongside it.
• Mutual information measures any dependence at all, linear or not, and is zero if and only if the variables are independent. It is harder to estimate from a sample and needs binning or a density estimate, but it is the honest general answer, and it gets its own lesson in the module on information.
• Plot the thing. A scatter plot takes five seconds and shows the U-shape, the outlier and the two clusters that every summary statistic hides. Anscombe's quartet — four datasets with identical means, variances and correlations and visibly different shapes — exists precisely to make this point.

Correlated features inside a model

Two consequences matter in practice.

Linear model coefficients become unstable. If two features are 0.98 correlated, the fit can put a large positive weight on one and a large negative weight on the other, with the pair nearly cancelling. Small changes to the data swing them wildly. The predictions may still be fine; the coefficients are not interpretable, and this is exactly the multicollinearity condition-number problem from the module on matrices.

Feature importance gets split arbitrarily. With two nearly identical features, a tree model uses one about half the time and the other the rest, so each shows roughly half the importance it deserves. Removing either changes nothing, which makes each look useless in an ablation. Always check the correlation structure before believing an importance ranking.

The one sentence about causation

Correlation is evidence about association, and association is consistent with causation in either direction, a common cause, selection, or chance. Establishing a causal direction needs something correlation cannot supply: an intervention, a randomised experiment, or a defensible assumption about the structure that generated the data. This is not a technicality to be waved away with a disclaimer. It is the reason A/B tests exist, and why a model trained on observational data can be an excellent predictor and a terrible guide to what to change.

The rule to keep

Correlation measures straight lines. Plot the data, compute Spearman as well as Pearson, and treat a zero as "no linear component found" rather than "no relationship".