Machine Learning, Foundations
The classical ground under modern AI, for someone who can read code.
- Level
- Some background helps
- Lessons
- 92
- Reading time
- 796 min
- Price
- Free, no sign-up to read
How a machine actually learns from data: what fitting means mechanically, why the features you choose matter more than the algorithm you pick, how to hold out data honestly, why accuracy lies on rare events, what the classic algorithms are good at, and how gradient descent and neural networks follow from all of it. You should be able to read code. You do not need calculus, statistics, or a maths degree — every idea arrives with real numbers attached.
Opens after the Getting Real Work Out of a Model exam
Sign in, finish that course, and pass its exam. You can read this syllabus meanwhile.
Go to Getting Real Work Out of a ModelModule 1
What learning is, and what it is not
Before any algorithm, the vocabulary and the honest framing. What a dataset actually is, what a loss function commits you to, what number your model has to beat before anyone should care, and the large class of problems where the correct machine learning solution is not to use machine learning.
By the end you can
Take a vague business question, state it as a dataset with a defined row, a defined target and a defined moment of prediction, name the loss and the baseline it must beat, and justify in one paragraph whether machine learning is the right tool at all
- 1What a machine actually learnsLearning means adjusting the parameters of a shape you chose until a chosen error number is as small as it can be.
- 2With answers, and withoutA supervised model learns your labels, not the reality behind them, so who made the labels is part of the model.
- 3What a row is, and why that decision is the whole projectDecide what one row is and when the target becomes knowable before you write any code, because every split, metric and deployment decision inherits that choice silently.
- 4The loss is where you state what a mistake costsSquared error fits the mean and fears outliers, absolute error fits the median and ignores them, log loss punishes confident mistakes without limit — and none of these is the metric you report.
- 5The number your model has to beat before anyone should careCompute the constant, persistence and existing-rule baselines before training, and report the gap between baseline and model rather than the model's score alone.
- 6Why a model that predicts well cannot tell you what to changeA predictive model reports associations that hold when you leave the world alone; changing something requires an experiment or an explicit causal assumption, and no accuracy score can substitute for either.
- 7The problems where the right answer is a ruleMachine learning earns its cost only when the decision rule is real but unwritable; when the rule is known, exact, or the data is too thin to tell a good model from a lucky one, code the rule.
- 8The whole loop, once, before we slow downThe whole discipline is nine steps with one irreversible arrow: the test set is touched once, at the end, after every decision has been made without it.
- 9A working setup that costs nothingThe standard toolchain for classical machine learning is free and open source; the ceiling on free compute constrains pretraining large models, not the tabular work that most jobs consist of.
Module 2
Linear models and the machinery of fitting
The simplest useful model, taken seriously. Fit a line by hand, watch an optimiser find the same answer, then extend it to probabilities, to many classes, and to curves — and learn to read the coefficients without believing more than they say.
By the end you can
Fit and diagnose a linear or logistic model end to end: compute its loss by hand on a handful of rows, tune the learning rate from the shape of the loss curve, state what a coefficient does and does not mean in the units of the data, and explain what ridge and lasso penalties change about the fitted parameters
- 10Fitting a line to five points, by handFitting means choosing parameter values that make a chosen error number small; the coefficient carries real units, and a linear model will extrapolate far outside its data without any signal that it is guessing.
- 11There is an exact answer, and we usually refuse itLinear regression with squared error has an exact one-step solution; gradient descent is taught because it is the only method that still works when the loss or the model has no such formula.
- 12Gradient descent, or walking downhill in fogGradient descent only knows the slope where it is standing; the step size decides whether that knowledge helps or destroys it.
- 13Batches, epochs, and the two knobs that decide everythingMini-batch size trades gradient noise against step count, and the learning rate must match the scale of the features — which is why standardising inputs fixes more convergence problems than raising the iteration limit.
- 14Diagnosing a model from the shape of its loss curvePlot training and validation loss on one axis every run: the gap diagnoses fit, the slope at the end diagnoses training length, and NaN or a flat line diagnoses a bug rather than a modelling problem.
- 15Logistic regression, built from odds rather than assumedLogistic regression models the log-odds as a straight line, which is why the sigmoid appears and why a coefficient multiplies the odds by a constant factor rather than shifting the probability by a constant amount.
- 16What a coefficient says, and the four ways it liesA coefficient is a change in the target per unit of one feature, holding the others in the model fixed — so it is only comparable when standardised, only stable when features are uncorrelated, and only meaningful relative to the exact set of columns included.
- 17Regularisation: paying for large coefficientsRidge and lasso add a price on coefficient size to the loss; lasso reaches exactly zero because the gradient of the absolute value stays constant as the coefficient shrinks, while ridge's push fades away near zero.
- 18Making a straight-line model bendA linear model bends by manufacturing columns — squares, interactions, bins, splines — and the interaction terms matter most, because finding those automatically is precisely what tree models do that linear models cannot.
- 19More than two classes, and what changesSoftmax makes classes compete for a fixed probability budget, which is correct when exactly one label is true and wrong for multi-label problems, where independent sigmoids with their own thresholds are needed.
Module 3
The data is the job
The part of machine learning that takes most of the time and wins most of the accuracy. Numbers, categories, missing values, dates and text turned into columns a model can use — and the two failures that quietly destroy more projects than any modelling mistake: leakage, and a sample that does not represent what you will predict on.
By the end you can
Take a raw table with mixed types, missing values, high-cardinality categories, dates and free text, and produce a fitted preprocessing pipeline that leaks nothing from the held-out data, handles categories unseen at training time, and whose every transformation you can justify by what the downstream model requires
- 20Features beat modelsAsk of every feature: at the moment I need the prediction, does this value already exist with this value?
- 21Scaling, and the transforms that change what a model can seeScaling is required by distance-based, penalised and gradient-fitted models and irrelevant to trees; the log transform is not cosmetic but a claim that ratios matter more than differences.
- 22Turning categories into numbers without lying about themOne-hot is the safe default because it asserts nothing about order or distance; target encoding is powerful and leaks unless computed out-of-fold and smoothed towards the base rate.
- 23When a column has thousands of levelsHigh-cardinality columns need a fixed-size representation — grouping, counting, hashing or a learned embedding — and any identifier that will not exist for an unseen row is not a feature at all.
- 24Missing values, and the information in the gapAdd a missingness indicator before imputing, because the fact that a value is absent is frequently more predictive than the value would have been — and let boosted trees learn their own direction for NaN rather than imputing at all.
- 25Outliers: error, extreme, or the thing you are looking forDecide first whether an extreme value is an error, a genuine rarity, or the target itself, because the same removal step is correct in the first case, harmful in the second, and destroys the problem in the third.
- 26Dates, durations, and the trouble with midnightDecompose timestamps into durations and cyclical components, use local time for behavioural features, and shift any rolling window by one period so it cannot include the value you are predicting.
- 27Text as columns: counting words before you embed themTF-IDF over word and bigram counts with a linear model is a strong, fast, fully interpretable text baseline; its ceiling is that it cannot represent word order or word similarity, which is precisely what embeddings add.
- 28Leakage: a catalogue of the ways the answer gets inLeakage is any feature whose training value would not be available in that form at prediction time, and the reliable test is to reconstruct a row using only information timestamped before the moment of prediction.
- 29Fit on train, transform everywhere: the contract that keeps you honestEvery transformation that learns a parameter must learn it inside each cross-validation fold, which is why a Pipeline changes the score you get rather than merely tidying the code.
- 30Where the rows came from, and who is missingA held-out set cannot detect a sample that misrepresents the population, because it is drawn from the same sample — so the composition of the data must be reasoned about from how it was collected, not measured from within it.
- 31When one class is 0.3% of the dataImbalance usually breaks the decision threshold rather than the model, so move the threshold and change the metric before resampling — and resample only inside the cross-validation fold, never before the split.
Module 4
Generalisation, and the discipline of held-out data
Why a model that fits your data is not yet a model that works, and the machinery for finding out: cross-validation, splits that respect the structure of the data, learning curves that tell you whether more data would help, and an honest account of what happens to a validation set you have used two hundred times.
By the end you can
Design a validation scheme that matches the structure of the data — grouped, stratified or time-ordered as required — run a hyperparameter search inside it without contaminating the estimate, and state from a learning curve whether the next improvement should come from more data, better features or a different model
- 32Train, validation, test, and the discipline of not lookingThe test set is a single-use measuring device; every peek turns it into another validation set.
- 33Overfitting, underfitting, and how to tellCompare training error with validation error; the gap between them, not the score itself, names the problem.
- 34Cross-validation: using every row twice, legallyCross-validation averages several held-out estimates so a score depends less on where the split fell, but the best score from a hyperparameter search is biased upward and needs an outer loop or an untouched test set to correct.
- 35Splits that respect the structure of the dataMatch the split to the structure — group by entity, order by time with a gap equal to the label delay, stratify small or imbalanced sets — and verify with a label-shuffle test that a correct pipeline collapses to chance.
- 36Bias and variance, with numbers attachedHigh bias is being wrong the same way every time and high variance is being wrong differently every time; the training-validation gap tells you which, and the classical U-shaped curve stops describing very large models.
- 37Would more data help? The curve that answers itPlot validation score against training-set size: a curve still climbing means collect more data, a converged pair at a poor score means the features or the model family are the limit, and no learning curve can anticipate a feature you have not built.
- 38Searching hyperparameters without fooling yourselfRandom search over log-scaled distributions beats grid search at equal cost because it tries many distinct values of the parameters that matter, and every configuration evaluated adds selection bias that only an untouched test set can measure.
- 39A validation set you have used two hundred timesThe best of many noisy validation estimates is biased upward by roughly the standard error times the square root of twice the log of the number of comparisons, so the count of experiments belongs in the report alongside the score.
- 40How much data do I need?Size the validation set from the precision you need — the standard error of a proportion at n=1,000 is about 1.1 points — and size the training set by events per feature rather than by total rows.
- 41Double descent, and the picture that stopped being completeTest error can fall again past the point where a model fits the training data exactly, because extra capacity gives the optimiser many perfect fits to choose among and gradient descent prefers smooth ones — an observation that is well replicated and not yet fully explained.
Module 5
Measuring a model honestly
Everything that turns a fitted model into a decision: the confusion matrix worked by hand, the threshold that nobody should leave at 0.5, curves that summarise every threshold at once, whether the probabilities mean anything, whether an improvement is real, and what fairness can and cannot be made to mean mathematically.
By the end you can
Given a fitted classifier and a stated cost of each kind of error, compute the confusion matrix at a chosen threshold, defend that threshold by expected cost, say whether the model's probabilities are calibrated, decide with an interval whether one model genuinely beats another, and report per-group performance with a defensible fairness criterion and its known trade-offs
- 42When accuracy liesChoose the metric that matches what each kind of mistake costs, then set the threshold yourself.
- 43The four numbers, worked throughPrecision divides by what the model said and recall by what was true, and precision alone is not a property of the model — it moves with the base rate of the population you run on.
- 44The threshold is a business decision, not a defaultThe model produces a score and the threshold converts it into a decision; choose the threshold from the relative cost of the two errors, the capacity to act, or a stated requirement — never from the library default.
- 45ROC and precision-recall: every threshold at onceAUC is the probability that a random positive outscores a random negative, so it measures ranking only and is insensitive to a large absolute number of false positives when negatives vastly outnumber positives — which is why imbalanced problems should be reported with average precision.
- 46Does 0.7 mean seventy per cent?Calibration asks whether a predicted 0.7 corresponds to 70% of cases occurring, is separate from ranking quality, and is fixed by a monotonic post-processing step that leaves AUC unchanged.
- 47Metrics for a number, and what each one hidesReport MAE and RMSE together because their gap measures how uneven the errors are, and plot residuals against predictions — the shape of the failure carries information no summary metric retains.
- 48Is that improvement real?Bootstrap the difference between two models on the same resampled rows and report the interval; a t-test across cross-validation folds violates independence and declares significance far too often.
- 49From a probability to a decision worth makingThe break-even threshold is the cost of a false positive divided by the sum of both error costs, it can and usually should be computed per row from that row's stakes, and the arithmetic is only valid for whoever's costs were put into it.
- 50Fairness as arithmetic: the definitions and what each one demandsDemographic parity, equalised odds and within-group calibration are three precise and different demands, dropping the protected column removes your ability to measure disparity without removing the disparity, and every one of these criteria is silent about bias already inside the labels.
- 51Why you cannot have all threeWhen two groups have different base rates, calibration within groups and equal error rates across groups are mathematically incompatible, so fairness requires an explicit and documented choice of which to satisfy.
- 52The average is hiding the failureA headline metric is a weighted average that hides its worst population, so slice by every dimension you can name, read fifty errors by hand to find the causes, and turn each failure mode into a fixed test set that runs on every change.
Module 6
Trees, ensembles, and the algorithms that win on tables
The family that actually wins most tabular problems, taken apart. How a single split is chosen, why averaging many deep trees works and why adding many shallow ones works for the opposite reason, the handful of hyperparameters that matter in the boosting libraries, and the two older algorithms still worth knowing — plus an honest account of what a feature importance number does and does not tell you.
By the end you can
Choose between a linear model, a random forest and a gradient boosting library for a given tabular problem and defend the choice, tune the four hyperparameters that actually matter in each, and explain why impurity-based feature importance is biased and what permutation importance and SHAP replace it with
- 53The classic algorithms, and when each is rightOn tabular data, use linear models as a floor and gradient boosting as the answer; everything else needs a stated reason.
- 54How a tree chooses where to cutA tree scores every candidate threshold by the weighted drop in impurity and takes the best one greedily, which gives it interactions for free but leaves it unable to express a diagonal boundary or extrapolate beyond the training range.
- 55Bagging: many unstable models, averagedBagging reduces variance by averaging, and a random forest adds per-split feature sampling because decorrelating the trees removes more variance than making each individual tree better.
- 56Boosting: each model fixes the last one's mistakesBoosting fits each shallow tree to the residual left by the previous ones and adds only a fraction of it, so accuracy accumulates slowly enough that noise averages out — which is why a boosted model needs early stopping and a forest does not.
- 57XGBoost, LightGBM, CatBoost: what actually differsThe three boosting libraries implement the same algorithm with different tree-growth and categorical handling, and six hyperparameters — learning rate, early stopping, depth, minimum leaf evidence, subsampling and L2 — cover essentially all the tuning that matters.
- 58k-nearest neighbours, and why distance stops meaning anythingk-NN makes the assumption behind most machine learning explicit — nearby points share targets — and the curse of dimensionality breaks it because distances concentrate, so the nearest neighbour in 1,000 raw dimensions is barely nearer than the farthest.
- 59Naive Bayes: wrong assumptions, useful resultsNaive Bayes assumes features are independent given the class, which is false and inflates its probabilities without usually changing which class ranks highest — so it discriminates well while being badly calibrated.
- 60Support vector machines and the kernel trickAn SVM maximises the margin and depends only on the support vectors; the kernel trick works because the optimisation needs only pairwise dot products, so an infinite-dimensional feature space costs one similarity computation instead of infinite coordinates.
- 61Feature importance, and the ways it misleadsImpurity importance favours high-cardinality features and is computed on training data; permutation importance on validation data measures what you meant, but both collapse when features are correlated, so permute clustered groups rather than single columns.
- 62Explaining one prediction, and the shape of one featureSHAP divides one prediction's departure from the average among the features with an exact additive decomposition, partial dependence shows a feature's average learned shape, and both assume feature independence in ways that break when the columns are correlated.
Module 7
Learning without labels, and learning a representation
What you can do when nobody has labelled anything: group similar rows, compress many columns into a few, find the rows that do not belong. Then the idea underneath modern search, recommendation and retrieval — an embedding, what it is geometrically, how to get one for free, and how to search a million of them in milliseconds.
By the end you can
Cluster a dataset and defend the number of clusters with more than an elbow plot, reduce dimensionality with PCA and state how much information the reduction discarded, explain what an embedding is as a geometric object, and build a working nearest-neighbour search over pretrained embeddings without a GPU
- 63k-means, and what it assumes about your datak-means minimises within-cluster squared distance, which assumes spherical clusters of similar size with no outliers and a known k, and it will return confident groupings from data that has no group structure at all.
- 64Choosing k, and why the elbow rarely helpsThe elbow rarely produces an unambiguous answer; use silhouette per cluster, the gap statistic — which can say that no clustering is right — and stability across subsamples, and accept that the useful k is often set by what the organisation can act on.
- 65DBSCAN and hierarchical clustering: when shape and structure matterDBSCAN defines clusters as dense regions, so it finds arbitrary shapes, decides the cluster count itself and labels outliers as noise — at the cost of assuming one density threshold suits the whole dataset, which HDBSCAN removes.
- 66PCA: fewer columns, and what you gave upPCA rotates to axes ordered by variance and keeps the first few, which requires standardised features and finds only linear structure — and because it never sees the target, the variance it discards can be the signal you needed.
- 67t-SNE and UMAP, and how they misleadt-SNE and UMAP preserve local neighbourhoods and deliberately distort cluster sizes and between-cluster distances, so the only reliable information in the plot is which points are neighbours — and they produce convincing clusters from pure noise.
- 68What an embedding actually isAn embedding maps things to vectors so that geometric closeness means similarity in the sense the training task defined, which is why the vectors are compared by angle rather than length and why they carry the training corpus's biases in their geometry.
- 69Building a working semantic search with no GPUFree CPU-sized embedding models plus an approximate nearest-neighbour index give working semantic search, but chunking affects quality more than model choice and pure embedding search fails on identifiers and negation, which is why production systems combine it with keyword search.
- 70Recommendation: learning a vector per user and per itemMatrix factorisation learns a vector per user and per item by fitting only the observed cells, which makes it an embedding trained on interactions — and its recommendations shape the next dataset, so offline metrics are always measured on data the previous model helped create.
- 71Anomaly detection, and the base rate that ruins itAnomaly detection models normality instead of learning a boundary, and at a 0.01% base rate even a 1% false positive rate produces a queue that is 99% wrong — so the design work is narrowing the population or ranking, not improving the detector.
- 72Self-supervised learning: making labels out of the data itselfSelf-supervised learning manufactures a supervised task from the data's own structure, and the pretext task chosen decides what the representation encodes — which is why a model pretrained with colour augmentation has learned that colour does not matter.
Module 8
Neural networks, built from what you already know
A network is a stack of linear models with a bend between them, trained by the same gradient descent from module 2. This module builds it that way: why one layer is not enough, how the chain rule assigns credit backwards, what actually stopped networks working before 2012, and the two architectures — convolution and attention — that carry almost everything you have heard of.
By the end you can
Explain why a multi-layer network with no non-linearity collapses to a single linear model, trace one backpropagation step through a two-layer network by hand, diagnose a training failure from the loss curve and the gradients, and fine-tune a pretrained model on a few hundred labelled examples using free compute
- 73Neural networks as the natural next stepA neural network learns its own features; the loss, the splits, the metrics and the thresholds do not change.
- 74XOR, and why the bend is the whole ideaA stack of linear layers collapses algebraically into one linear layer, so every capability a network has beyond linear regression comes from the non-linearity applied between the layers.
- 75The bends: ReLU, and why sigmoid stopped being used inside networksSigmoid's derivative peaks at 0.25, so backpropagation shrinks the gradient by at least a factor of four per layer and deep networks stopped learning; ReLU's derivative is exactly 1 where it is active, which is why a crude hinge replaced a smooth curve.
- 76Backpropagation, traced through one tiny networkBackpropagation multiplies the gradient arriving from above by each operation's local derivative, which computes every weight's gradient in one backward pass — and explains why gradients vanish, why dead ReLUs stop learning, and why training needs far more memory than inference.
- 77Starting well, and staying well-behavedInitialisation scale keeps activation variance stable across layers, normalisation keeps it stable across training, and a residual connection gives the gradient an unmultiplied path backwards — which is what made networks deeper than about twenty layers trainable at all.
- 78Momentum, Adam, and what the optimiser is doing for youMomentum averages recent gradients so oscillations cancel and consistent directions accumulate; Adam adds a per-parameter step size from the average squared gradient, and its weight decay must be applied outside that scaling, which is what AdamW does.
- 79Keeping a large network from memorisingEarly stopping and more data outrank every other regulariser, dropout must be disabled at inference or predictions become random, and before regularising anything you should confirm the network can deliberately memorise twenty examples — otherwise you are tuning around a bug.
- 80Convolution: the same detector, everywhere in the imageA convolution applies the same small filter at every position, so it learns a pattern once instead of separately per location — which cuts parameters by orders of magnitude and builds a hierarchy from edges to objects as depth increases.
- 81Sequences: from recurrence to attentionAttention lets every position read every other in one step by computing a similarity-weighted average, which removes the multiplication chain that limited recurrent networks and parallelises training — at a cost that grows with the square of the sequence length.
- 82Standing on somebody else's computePretrained early layers encode general structure that transfers across tasks, so start by freezing the backbone and fitting a linear model on its features — and escalate to full fine-tuning only when the measurement says your data supports it.
Module 9
Into the world, and keeping it honest
The half of the job that starts when the notebook closes. Serving a model without the training and serving paths drifting apart, monitoring the things that actually fail, deciding when to retrain, making a model small enough to be affordable, and the two obligations that outlast the model: not letting it corrupt its own training data, and documenting who it was built for.
By the end you can
Take a fitted model to a running service with a versioned artefact, a reproducible training run, monitoring that distinguishes covariate shift from concept drift, a stated retraining trigger, and a written record of what the model was trained on and who it should not be used for
- 83The model is about 5% of what you have to buildThe model is a small fraction of a production system; the rest is feature computation, versioning, serving, monitoring and rollback — and knowing the serving constraint before choosing a model is what stops you building something that cannot be deployed.
- 84Batch, online, and the latency budgetFeature retrieval usually dominates the latency budget rather than model inference, so precompute slow-moving features — and run a new model in shadow mode against real traffic before it serves anyone.
- 85When training and serving quietly disagreeTraining-serving skew is a systematic difference between how a feature is computed offline and online, and the reliable defences are shipping one serialised pipeline, importing shared feature code rather than copying it, and an automated test asserting both paths produce identical predictions on the same rows.
- 86Monitoring: what to watch, and in what orderMonitor system health, input distributions, prediction distributions and finally actual performance — the first three need no labels and move earlier, and the alert must name which kind of shift it found because covariate, label and concept drift take different remedies.
- 87When to retrain, and what breaks when you doRetraining is a deployment and needs the same gates as one, including re-choosing the decision threshold for the new score distribution — and it is the correct remedy only when the relationship has genuinely changed, not for upstream data faults or serving bugs.
- 88Being able to build the same model twiceReproducibility requires pinning code, data, environment and seeds together, and because GPU and parallel arithmetic are not bitwise deterministic, the standard to hold is a rerun scoring within noise of the original rather than an identical artefact.
- 89AutoML: what it automates, and what it cannotAutoML automates model selection and hyperparameter search — genuinely well, and free — but it cannot frame the problem, build features, detect leakage or choose the metric, which is where most of a model's quality is determined.
- 90Making a model small enough to be affordableQuantisation, distillation and pruning trade a small amount of accuracy for large reductions in size and latency — and unstructured pruning reduces parameter count without reducing runtime, which is the distinction most often missed.
- 91When the model changes the data it will be trained onA deployed model shapes the data its successor is trained on, so offline metrics measure agreement with the previous model's choices — and only randomised exploration, logged propensities and control groups produce evidence that escapes the loop.
- 92What you owe the people in the dataA model card recording the target definition, training population, per-slice performance and out-of-scope uses costs an hour and outlives everyone involved — and the legal questions around copyright, erasure and automated decisions are genuinely unresolved and differ by country.