Probability once you know something

The bar means "given"

P(A | B) is the probability of A once you know B is true. It is not a new kind of probability; it is the same fraction computed over a smaller universe. Formally:

P(A | B) = P(A and B) / P(B)

Read the denominator as the shrinking. You have thrown away every case where B is false, so you divide by how much of the world is left.

Worked, with a table you can count

A hundred support tickets. Sixty are billing, forty are technical. Of the billing tickets, 45 are resolved on the first reply; of the technical tickets, 20 are.

P(billing) = 60/100 = 0.60
P(first-reply resolved) = (45 + 20)/100 = 0.65
P(billing and resolved) = 45/100 = 0.45

P(resolved | billing) = 45/60 = 0.75
P(billing | resolved) = 45/65 = 0.69

Note that P(resolved | billing) = 0.75 and P(billing | resolved) = 0.69 are different numbers. They answer different questions, and swapping them is the most common error in applied probability. It has a name — the prosecutor's fallacy — and it appears constantly in reasoning about model outputs: "the model flags 95 per cent of fraud" is P(flagged | fraud), and it tells you almost nothing about P(fraud | flagged), which is the number you act on.

The multiplication rule, and how a language model works

Rearranging the definition gives

P(A and B) = P(B) * P(A | B)

which extends to any number of events by chaining:

P(A, B, C) = P(A) * P(B | A) * P(C | A, B)

This chain is the entire probabilistic content of a language model. The probability of a sentence is

P(w1, w2, ..., wn) = P(w1) * P(w2 | w1) * P(w3 | w1, w2) * ...

Every forward pass computes one factor: the distribution over the next token given everything before it. Generation multiplies them out by sampling one at a time. That is why context matters mechanically rather than metaphorically — the conditioning set is literally the input to the function.

It is also why probabilities of long sequences are computed as sums of logs. Multiplying 500 numbers each below 1 underflows to zero in any floating-point format, a point returned to in the module on logs.

Independence, defined properly

A and B are independent when knowing one tells you nothing about the other:

P(A | B) = P(A)      equivalently      P(A and B) = P(A) * P(B)

Independence is a strong claim and it is usually false in the data you have. Two support tickets from the same customer on the same day are not independent. Two frames of a video are not independent. Two sentences from the same document are not independent.

This matters more than it sounds, because nearly every statistical formula you will use — standard errors, confidence intervals, significance tests — assumes independent samples. When your samples are clustered, the effective number of independent observations is far below the row count, and every interval you compute is too narrow. A test set of 10,000 sentences drawn from 50 documents does not give you 10,000 independent observations; it gives you something closer to 50.

The base rate, which is the part people drop

Here is the pattern to watch for. Somebody reports "the classifier is 95 per cent accurate on fraudulent transactions". You want to know whether to act on a flag. The missing number is how common fraud is at all.

If fraud is 0.1 per cent of transactions, then out of 100,000 transactions there are 100 fraudulent ones. The classifier catches 95 of them. But it also fires on some fraction of the 99,900 legitimate ones — at even a 1 per cent false-positive rate, that is 999 false alarms. So of 1,094 flags, 95 are real: about 9 per cent.

A 95-per-cent-accurate detector that is wrong 91 per cent of the times it fires. Nothing is broken. The base rate did that, and the next lesson makes the arithmetic general.

The rule to keep

P(A | B) and P(B | A) are different numbers, and the one you are quoted is usually not the one you need. Ask what the denominator is, every time.