What a float32 actually stores, and where its seven digits run out

Thirty-two bits, three fields

A float32 is scientific notation in binary. Its 32 bits are split into a sign (1 bit), an exponent (8 bits) and a fraction called the mantissa (23 bits):

value = (−1)^sign × 1.mantissa × 2^(exponent − 127)

The leading 1 is not stored, because in binary the first significant digit is always 1, so the mantissa carries 24 bits of precision for the price of 23. Everything about how floats behave follows from those three field widths.

Precision: about seven digits

Twenty-four binary digits is 24 × log10(2) = 7.2 decimal digits. The gap between 1 and the next representable number is 2^-23 = 1.19 × 10^-7, called machine epsilon. Any number is stored to a relative accuracy of about half that, six parts in a hundred million. The absolute gap between neighbouring floats grows with the number: near 1 it is 10^-7, near 1,000 it is 10^-4, near a million it is 0.06, and past 2^24 = 16,777,216 it is 2.

That last fact bites. A float32 counter incremented by 1 stops at 16,777,216, because 16,777,217 is not representable and the addition rounds back down:

import numpy as np
x = np.float32(16_777_216)
print(x + np.float32(1))       # 16777216.0

A running total of tokens, steps or samples kept in float32 silently stalls at seventeen million. Keep counters as integers.

Range: about 10^±38

Eight exponent bits give exponents from −126 to +127, so the largest float32 is about 3.4 × 10^38 and the smallest normal one about 1.2 × 10^-38. Below that, the format gives up precision to reach a little further, down to 1.4 × 10^-45, after which the number is zero. Module 5 used the top of that range: exp(x) overflows to infinity at x = 88.7. The bottom matters too. A probability of 10^-40 is representable only as a denormal with a few bits of precision, and the product of two probabilities of 10^-25 each is exactly zero. This is the concrete reason everything probabilistic is done in log space.

Why 0.1 + 0.2 is famous, and why the format matters

One tenth is not representable in binary, any more than one third is in decimal. Stored as float64, 0.1 is really 0.1000000000000000055, and

0.1 + 0.2 == 0.3          # False in float64: 0.30000000000000004

In float32 both sides happen to round to the same number, 0.300000012, and the comparison is True. The lesson is not about 0.3. It is that equality between floats is an accident of rounding, and a test like if loss == 0.0 or assert a == b on computed values will pass or fail depending on the format, the order of operations, and the hardware. Compare with a tolerance:

np.isclose(a, b, rtol=1e-5, atol=1e-8)

and choose the tolerances from the format: rtol near 10^-5 for float32, 10^-12 for float64.

The gap you should picture

Representable floats are not evenly spaced. They are dense near zero and sparse far from it, with the same number of floats between 1 and 2 as between 1,024 and 2,048. This has one consequence you will meet constantly: adding a small number to a large one loses the small one. 10^8 + 1 in float32 is 10^8, because the gap between floats near 10^8 is 8. The next lesson but one is about the damage that does.

Doubles, and when to use them

Float64 has 11 exponent bits and 52 mantissa bits: about 16 digits and a range of 10^±308. NumPy and Python use it by default; deep-learning frameworks use float32 by default because it halves memory and the arithmetic units are faster. The right rule: float32 for the model, float64 for anything that accumulates, checks, or is compared. A loss averaged over a million batches, a gradient check, a statistic computed by subtraction: float64. A weight matrix: float32 or smaller, and the next lesson says how much smaller.

Reading the bits yourself

import struct
bits = struct.unpack(">I", struct.pack(">f", 0.1))[0]
print(f"{bits:032b}")     # 0 01111011 10011001100110011001101

Sign 0, exponent 01111011 = 123, so 2^(123 − 127) = 2^-4 = 1/16, mantissa 1.6000000238..., and 1.6 / 16 = 0.1000000015. The repeating 1001 in the mantissa is the binary expansion of one tenth being cut off at 23 bits, and that cut is where the 0.0000000015 came from. Once you have seen it, every rounding surprise in the rest of this module has an address.