How variance travels through a layer, and the initialisation that follows

One unit's output is a sum

A unit in a linear layer computes y = Σ w_i x_i over fan_in inputs. Treat the weights and inputs as random: each w_i drawn with variance σ_w², each x_i with variance σ_x², all independent and centred on zero. Module 4's rule that variances of independent terms add, and that the variance of a product of independent centred variables is the product of the variances, gives

Var(y) = fan_in × σ_w² × σ_x²

That one line decides whether a deep network can be trained from a random start.

What happens over twenty layers

Suppose you initialise every weight with a "small random" standard deviation of 0.05, a common default in hand-written code, and the layer has fan_in = 1024. Then

Var(y) / Var(x) = 1024 × 0.0025 = 2.56

Each layer multiplies the variance by 2.56. A ReLU after it zeroes half the values and roughly halves the variance, so the net factor is about 1.28 per layer. After twenty layers the activations have variance 1.28^20 ≈ 139 times the input's. After sixty, 10^6. The final logits are enormous, the softmax saturates, and module 5's gradient p − y is either 0 or ±1 everywhere: the model begins its life confidently wrong and with no useful gradient.

Try 0.01 instead:

1024 × 0.0001 × 0.5 = 0.051 per layer

After twenty layers the variance is 10^-26. Every activation is effectively zero, every gradient with it, and by forty layers the values are below anything float32 can represent. The model does not learn because there is no signal to learn from. Module 3's vanishing gradients, from the other end.

The value that keeps it level

Set the per-layer factor to 1:

fan_in × σ_w² × ½ = 1    →    σ_w = √(2 / fan_in)

For fan_in = 1024, σ_w = 0.044. Between the two failures above sits a single number that keeps the variance constant through every layer, and it depends only on the width of the layer. That is the He, or Kaiming, initialisation, and the 2 is the ReLU's halving. Without a ReLU, or with tanh, the factor is 1/fan_in, the Xavier or LeCun rule. machine-learning-foundations states these rules; this is where they come from, and the derivation is short enough that you could rediscover it if you forgot.

The backward pass has the same arithmetic

Gradients flowing backward through the layer are also sums, over fan_out terms this time, so the same argument gives σ_w = √(2 / fan_out) to keep gradient variance level. The two demands conflict unless fan_in = fan_out; the usual compromise averages them, and the disagreement is small enough not to matter for layers of similar widths. It matters for a layer that maps 4,096 dimensions to 32: initialise for the forward pass and the backward variance is off by a factor of 128 in that layer alone.

Why normalisation layers were the other answer

Initialisation gets the variance right at step zero. Training moves the weights, and nothing then holds the variance in place. Layer normalisation and batch normalisation re-standardise the activations at every step, subtracting the mean and dividing by the spread, which enforces Var = 1 by construction regardless of what the weights have become. They are the runtime version of the calculation above, and the reason very deep networks became trainable. Residual connections are a third answer: adding the input straight through means the variance grows only additively per block rather than multiplicatively, and 20 × small is survivable where 1.28^20 is not.

Check it in three lines

import numpy as np
rng = np.random.default_rng(0)
x = rng.standard_normal((1024, 512))                    # 512 examples, 1024 wide
for sigma in (0.05, 0.01, np.sqrt(2 / 1024)):
    h = x.copy()
    for _ in range(20):
        W = rng.standard_normal((1024, 1024)) * sigma
        h = np.maximum(W @ h, 0)                        # ReLU
    print(sigma, h.std())                               # huge, ~0, ~1

The first run's standard deviation is in the tens, the second's is around 10^-13, and the third's is about 1. Twenty lines of NumPy reproduce the whole argument, and the same script, pointed at your own architecture, tells you whether its initialisation is sane before you spend a GPU-hour finding out.

What the argument assumed

Independence between weights and inputs, which holds at initialisation and fails once training correlates them. Zero means, which a ReLU breaks by producing only non-negative outputs, so the halving is approximate. And a linear layer; convolutions have fan_in = channels × kernel area, attention has its own scaling, the 1/√d in module 1's dot-product lesson, which is the same variance argument applied to a dot product of two d-dimensional vectors. The idea survives all of these. The constant changes.