Sparse and dense, and the memory each costs

Two ways to write down a document

Suppose your vocabulary has 50,000 words and you want to represent the sentence "the train was late".

The sparse way. Make a vector with 50,000 slots, put a count in the slot for each word present, and zeroes everywhere else. Four non-zero entries out of 50,000. Stored naively that is 50,000 numbers; stored sensibly it is four (index, value) pairs, because the zeroes carry no information and need not exist.

The dense way. Run the sentence through an embedding model and get 768 numbers, almost none of them zero, none of them individually interpretable.

Neither is a better idea in general. They record different things. The sparse vector says exactly which words are here. The dense vector says what this is roughly about.

One-hot, the simplest sparse form

The atom of sparse representation is one-hot encoding: a vector with a single 1 and zeroes elsewhere. Word 8,417 in a 50,000-word vocabulary becomes a vector with 1 in position 8,417.

Its properties are worth stating because they explain what came next. Every one-hot vector has length 1. The dot product between any two different one-hot vectors is 0, so every word is exactly as unrelated to every other word — cat and dog are as distant as cat and parliament. And the space needed grows with the vocabulary, one dimension per word.

The embedding table fixes all three at once. Multiply a one-hot vector by a 50,000 × 768 matrix and you select row 8,417 of that matrix. That is the entire mechanism: an embedding lookup is a one-hot multiplication that somebody optimised into an array index. The rows are learned, so related words end up with related rows, and the representation is 768 numbers instead of 50,000.

The memory arithmetic

Do the sums, because they decide architecture.

One million documents, dense 768-dimensional embeddings, 4-byte floats:

1,000,000 x 768 x 4 bytes = 3,072,000,000 bytes = 3.07 GB

That fits in the RAM of an ordinary laptop, which is why in-memory vector search is practical at this scale. At ten million documents it is 30.7 GB and you are buying a server or quantising. Store the same vectors as 1-byte integers instead of 4-byte floats and it falls to 768 MB, a point returned to in the module on quantisation.

The same million documents, sparse, averaging 120 distinct terms each, stored as an index plus a value at 8 bytes per pair:

1,000,000 x 120 x 8 bytes = 960,000,000 bytes = 0.96 GB

Less than the dense version, and it grows with document length rather than with vocabulary size. An inverted index — the structure behind every keyword search engine since the 1970s — makes this smaller still and queries very fast, because you only ever touch the postings lists for the words in the query.

TF-IDF: making sparse counts behave

Raw counts are a poor sparse representation because common words dominate. The classical repair is TF-IDF: multiply each term's count in a document by the inverse document frequency,

idf(t) = log( N / df(t) )

where N is the number of documents and df(t) how many contain the term. A word in every document gets log(1) = 0 and vanishes. A word in one document in a thousand gets log(1000) = 6.9 and dominates. BM25, the default in Lucene, Elasticsearch and SQLite's FTS5, is a refined version with saturation on term frequency and a correction for document length.

This is worth knowing because BM25 is a strong baseline that costs nothing to run and is genuinely hard to beat on queries containing rare exact terms — a part number, a surname, an error code, NullPointerException. A dense model has to have learned that token to represent it; a sparse index simply matches it.

Why real systems use both

The failure modes are complementary, which is the whole argument for hybrid search.

• Sparse fails on synonyms and paraphrase. A query for "cheap flights" does not match a document saying "budget airfare" if no word overlaps.
• Dense fails on rare exact strings. Ask for order AX-99182 and the embedding has no idea; the token was probably never seen.

Hybrid search runs both and combines the rankings, typically with reciprocal rank fusion, which merges by rank position rather than by score and so avoids the problem that the two systems' scores are on incompatible scales. It reliably beats either alone on mixed query workloads.

The free path

None of this needs a paid service. scikit-learn gives you TfidfVectorizer in one line. SQLite ships FTS5 with BM25 built in and runs on a phone. rank_bm25 is a small pure-Python implementation. For dense vectors, sentence-transformers runs a compact model on a CPU, and FAISS or hnswlib index them locally. A hybrid search system over a few hundred thousand documents runs on a laptop with no account anywhere.

The rule to keep

Sparse records identity, dense records similarity, and the memory arithmetic tells you which is affordable at your scale. If your queries contain names, codes or numbers, you need the sparse half no matter how good the embedding model gets.