Power laws, and the straight line on a log-log plot

Two shapes that look alike and are not

An exponential, y = c × b^x, grows or decays by a fixed factor for each fixed step in x. A power law, y = c × x^k, changes by a fixed factor for each fixed ratio in x: doubling x always multiplies y by 2^k, whether x goes from 1 to 2 or from a million to two million. Both curve on ordinary axes, and both are easily mistaken for each other by eye.

Logs separate them. Take logs of the power law:

log y = log c + k × log x

That is a straight line in log x with slope k. So a power law is a straight line on log-log axes, and the exponent is the slope you can read off with a ruler. An exponential is a straight line on log-linear axes, y logged and x not. Plot the data both ways; whichever is straight tells you which law you have.

Reading the slope

Two points on a log-log plot, (10, 500) and (1000, 5):

k = (log 5 − log 500) / (log 1000 − log 10)
  = (0.699 − 2.699) / (3 − 1)
  = −2 / 2 = −1

So y ∝ 1/x. The base of the log does not matter, since it cancels in the ratio. Any two points far enough apart give the slope; use points at least a decade apart, because near points make the slope hostage to noise.

A slope is a promise about doubling. A slope of −0.5 says that quadrupling x halves y, since 4^(−0.5) = 0.5. A slope of −0.05 says doubling x cuts y by 2^(−0.05), a factor of 0.966: three and a half per cent. Small exponents are the mathematical form of diminishing returns, and they are what the loss-versus-compute curves in how-llms-work look like. Read the slope before you read the headline.

Zipf, and why the vocabulary has a tail

Rank the words of a large English text by frequency. The most common, "the", is about 7 per cent of all words. The second is about half that, the tenth about a tenth, the hundredth about a hundredth. Frequency is roughly proportional to 1 / rank, a power law with exponent −1, and it holds across languages and across centuries of text. This is Zipf's law.

It has consequences you have met. A tokeniser gives short, single tokens to common words because they pay for themselves, and splits rare words into pieces because a vocabulary slot for each would be wasted; that is the head of the distribution being served and the tail being approximated. It also means the tail is enormous: with exponent −1, half of all the distinct words in a corpus appear once, and no amount of data gets the model many examples of any one of them. Rare-word failures are not a bug in a particular model. They are the arithmetic of the distribution the model was trained on.

Power laws and heavy tails

The heavy-tailed distributions of module 4 are usually power laws in their tails. A Pareto distribution with tail exponent α has infinite variance when α ≤ 2 and no finite mean at all when α ≤ 1. City sizes, wealth, file sizes, the number of followers an account has, and the length of the longest request in a day all sit in this family, which is why their averages misbehave and their percentiles do not. When you see a straight line on a log-log histogram, expect the mean to be unreliable, and expect the largest observation to grow as you collect more data rather than settling down.

Where the straight line lies

A straight line over one or two decades is weak evidence of a power law. A log-normal distribution looks straight on log-log axes over a limited range; so do several others. The distinguishing region is the far tail, which is exactly where you have the fewest points. Three specific cautions:

• Do not fit the exponent by least squares on a log-log histogram. Binning and logging make the noise unequal across the line, and the fit is biased. The maximum-likelihood estimate for a tail exponent from n values above a threshold x_min is one formula: α = 1 + n / Σ ln(x_i / x_min). It takes one line and it is what the powerlaw package computes.
• Do not extrapolate the line. A slope measured between 10^3 and 10^5 says nothing about 10^7. The curves you will most want to extrapolate, of loss against scale, are the ones with the fewest points at the far end.
• Check the exponent's uncertainty. With twenty points, the slope carries an error bar of several tenths, and a slope of −1.1 against −0.9 is not a distinction the data can make.

In code

import numpy as np, matplotlib.pyplot as plt
plt.loglog(x, y, "o")                        # straight means power law
k, logc = np.polyfit(np.log(x), np.log(y), 1) # slope k; fine for a curve, not a histogram

The polyfit on logs is acceptable for a smooth curve such as a loss against dataset size, where each point is a measurement rather than a count. For a histogram of a heavy-tailed quantity, use the likelihood formula. Both are free, and both fit on a laptop. The skill being trained is the one you use every time a paper shows a straight line on log axes: find the slope, find its uncertainty, and ask what the line is being asked to promise beyond the data it was drawn through.