Building With AI
From your first API call to a feature you can trust
- Level
- Some background helps
- Lessons
- 80
- Reading time
- 701 min
- Price
- Free, no sign-up to read
You can write some code. You have used a chat model. Now you want to put one inside something real — a support triage tool, a search box that answers questions, a feature your users hit a thousand times a day — and you have found that the gap between a good demo and a working feature is wide. This course closes that gap. It covers the HTTP call underneath everything and what it costs, how system prompts really behave, structured output and its limits, tool use and who is actually running the code, retrieval done properly, chunking and embeddings with real numbers, what an "agent" is once the marketing is removed, how to evaluate your own feature instead of guessing, and the failure modes to design for before launch. Every lesson has code you can run. The course is honest about what does not work: schemas do not make answers true, retrieval does not stop hallucination, and most agent demos fail for reasons no bigger model will fix. It does not cover training or fine-tuning models. This is about building with them.
Opens after the Python, From Zero, For AI exam
Sign in, finish that course, and pass its exam. You can read this syllabus meanwhile.
Go to Python, From Zero, For AIModule 1
The call underneath everything
One HTTP request, priced in tokens. What you send, what comes back, what it costs, what it does when it fails, and how to run the same thing on a laptop with no GPU.
By the end you can
Send the same request to a hosted model and to a model running on your own machine, read every field of the response including token counts and stop reason, and predict within a factor of two what a proposed feature will cost per thousand users before you write it
- 1The call underneath everythingThe API is stateless: every turn you pay to resend the entire conversation.
- 2System prompts, user turns, and who the model listens toA system prompt is the strongest thing you say, not a rule the model cannot break.
- 3Tokens, and why a Hindi bot costs three times an English oneCost, limits and latency are all measured in tokens, and the same sentence in Hindi can cost three times what it costs in English because the vocabulary was fitted to English.
- 4Temperature, sampling, and why the same input gives a different answerSampling settings live in your code, not the model, and temperature 0 still varies because floating-point reductions and shared batching change which of two near-tied tokens wins.
- 5Streaming, and the difference between fast and feeling fastStreaming does not make generation faster; it replaces total latency with time-to-first-token as the number the user feels, and it costs you the ability to validate or moderate before the text is on screen.
- 6Every field in the response, and the two that catch bugsCheck stop_reason before you parse and log the returned model string and usage on every call, because a truncated answer and a silently swapped model version both look like a working response until they do not.
- 7Errors, retries, and how a retry loop takes down your own serviceRetry only what a second attempt could fix, with jittered backoff under a wall-clock deadline and a concurrency cap, or your retry logic will convert a slow provider into your own outage.
- 8Choosing a model, and designing so you can change your mindThe price gap between model tiers is 30-100x, benchmarks do not transfer to your task, so route cheap-first against twenty of your own cases and keep the provider behind one narrow interface.
- 9Running a model on a laptop with no GPUA quantised 7B model is about 4 GB, runs on an ordinary laptop through an OpenAI-compatible endpoint on localhost, and is close to hosted quality on classification, extraction and rewriting while clearly behind on multi-step reasoning.
- 10Sending images, PDFs and audio, and what they costImages cost roughly width times height over 750 in tokens, PDFs need a text-layer-then-render fallback, and every extracted value should carry a verbatim quote you can find in the source, because a model asked to read something illegible will invent something plausible instead of refusing.
Module 2
Prompting as engineering
A prompt in production is not a clever sentence. It is a versioned interface with inputs, a contract, a cache strategy, a defined behaviour when it has no answer, and a test suite.
By the end you can
Turn a prompt that works once into a versioned, cached, example-driven interface with a stated behaviour when the model does not know, and justify with cost and latency numbers whether a given task needs a longer prompt, a chain of smaller ones, a reasoning budget, or a fine-tune
- 11A prompt is an interface, not a paragraphWrite rules you could assert on, always define an explicit sentinel for I-do-not-know, and prune the prompt by deleting lines and re-running your cases, because unread instructions still bill on every call and compete with the ones that matter.
- 12Examples that teach, and examples that misleadChoose examples that cover the boundary rather than the typical case, balance the labels, and remember that examples fix shape and consistency but never fix missing knowledge.
- 13One big prompt, or four small onesSplit a prompt when the pieces need different models, different tests or a checkable intermediate, put a deterministic check between every link, and stop at three or four steps because accuracy multiplies down the chain.
- 14Reasoning models, thinking budgets, and when they are a waste of moneyExtended thinking is the model buying itself more forward passes by emitting more output tokens, so it helps when the difficulty is deciding the answer and not when the difficulty is knowing a fact or producing a format.
- 15Prompt caching, and the ordering rule that pays for itselfPrompt caching matches an exact byte prefix from position zero, so every static part must come before every variable part, and one timestamp near the top destroys the entire discount.
- 16Making the model quote before it answersRequire verbatim quotes before the answer and check in code that each quote appears in the passage it cites, which converts fabrication from invisible to detectable without claiming the answer is therefore true.
- 17Scope, refusals, and a voice that stays putDefine scope as an explicit list of what the assistant does not do plus what to offer instead, express tone as constraints you could assert on, and test over-refusal as carefully as you test refusal.
- 18Prompts belong in version controlA prompt is source code that ships behaviour, so version it in files, render it through a strict template that escapes your own delimiters, log the rendered bytes with the model string, and diff two versions over the same cases before switching.
- 19Building for users who do not write in one languageReply in the user's script and not merely their language, translate the query for retrieval rather than translating your instructions, and score quality separately per language because an average hides the languages you are failing.
- 20When prompting stops working: the fine-tuning decisionFine-tuning teaches format, style and compression, never facts, so ask first whether the model can do the task when shown the answer, and treat a tuned model as a permanent dependency rather than a one-off project.
Module 3
Structure, tools and the loop
Getting a machine-readable answer out, giving the model things it can ask you to run, and building the loop that ties them together without letting it run away.
By the end you can
Design a schema and a tool surface a model can actually follow, write an agent loop with a step budget and a validated boundary at every hop, and say for each failure in that loop whether it should be retried, escalated to a human, or must stop the run
- 21Structured output, and what a schema cannot promiseSchemas guarantee shape, never truth: a perfectly valid object can be perfectly wrong.
- 22Designing a schema a model can actually fillFlat, enum-heavy, explicitly nullable schemas with units in the field names and rules in the descriptions succeed where deeply nested ones fail, and constrained decoding guarantees the shape of a wrong answer just as firmly as a right one.
- 23Extraction that runs on ten thousand documentsLocate the region before extracting from it, verify repeating rows with arithmetic rather than prompting, and design the review queue and its override log as the actual product, because the corrections are your evaluation set and your training data.
- 24Tool use: the model never runs anythingThe model can only ask for a tool call; your code runs it, so your code must authorise it.
- 25Designing tools a model can use correctlyTool descriptions are the model's only interface, so shape tools around user questions rather than API endpoints, say explicitly when not to use each one, and never let identity be a parameter the model can choose.
- 26What an agent actually is, and why the demos failAn agent is a loop with a step budget; reliability multiplies, so short chains beat clever ones.
- 27Writing the loop, with the stopping conditionsAn agent loop needs three independent budgets — steps, cost and wall clock — because context growth makes cost quadratic in run length, and it must sort every failure into recoverable, retryable or fatal, since handing a permission denial back to the model turns it into a search for a way around your access control.
- 28MCP: what a tool protocol buys you, and what it costsMCP standardises tool discovery so capabilities become a deployment concern rather than a code change, at the cost of a new trust boundary: server-supplied descriptions enter your prompt and can change after you approved them.
- 29Letting a model run code without letting it run your machineA code tool is a shell rather than a tool, so isolate it in a disposable container with no network and no credentials, and always show the generated code beside the result because inspectable working is the only real check on it.
- 30Multiple agents, and the honest costThe real benefit of multiple agents is context isolation and parallelism on genuinely independent subtasks, paid for with roughly an order of magnitude more tokens, lossy summaries at each boundary and ordinary concurrency bugs.
- 31Approval gates, undo, and handing over to a personGate on reversibility rather than importance, put the gate in the tool where the model cannot argue past it, and watch the approval and override rates because a gate approved 99% of the time is theatre.
Module 4
Retrieval and the knowledge problem
A model knows what was in its training data and what is in its context window, and nothing else. This block is about filling the window well: what an embedding actually measures, which model to use and how to test it, where to store vectors and when you need no database at all, why keyword search is still half the answer, reranking, rewriting the question, the ingestion work that decides everything, permissions and freshness, and when a long context window makes the whole pipeline unnecessary.
By the end you can
Build a retrieval pipeline from raw documents to a ranked set of chunks — embed, index, fuse with keyword search, rerank, filter by permission — measure its recall on a gold set of your own questions, and say from cost and quality numbers whether a given corpus should be retrieved or simply placed in the context window
- 32Retrieval: what it fixes and what it does notRetrieval decides what the model can know; when RAG fails, it is usually search that failed.
- 33What an embedding is, and what "close" actually meansAn embedding's closeness means only what the model's training pairs meant by closeness, so similarity thresholds are model-specific, exact identifiers blur, negation nearly vanishes, and calibrating on your own data is the step that cannot be skipped.
- 34Chunking and embeddings in practiceEmbeddings rank what a chunk is about, not whether it answers you, so rerank before trusting the top hit.
- 35Choosing an embedding model: dimensions, languages and the leaderboard trapShortlist embedding models from the leaderboard but choose on recall over your own fifty queries, and check the embedder's context limit before anything else, because a chunk longer than that limit is silently cut and the cut half is never searched.
- 36Vector search: when NumPy is enough, and what an index trades awayBelow about a million vectors a brute-force dot product on a CPU is exact and fast enough, and above it an approximate index buys speed by giving up a measurable fraction of recall, which you must measure against exact results rather than assume away.
- 37Hybrid search: why keyword search is still half the answerEmbeddings blur exact strings and keyword search cannot see paraphrase, so a production search runs both and fuses them by rank rather than score, because BM25 and cosine scores live on scales that cannot be added.
- 38Reranking, and how many chunks to actually sendA cross-encoder reads the query and passage together and so can judge whether the passage answers the question, which a bi-encoder's separate vectors never can, and the number of chunks you then send is a measured trade between recall, distraction and tokens per query rather than "as many as fit".
- 39Rewriting the question before you search itThe text a user typed is rarely the best search query, because follow-up turns omit the topic, terse questions omit the vocabulary and mixed-language questions miss the corpus, so a cheap rewriting call before retrieval — condensing, expanding, decomposing or extracting filters — is usually the largest single improvement available.
- 40Ingestion: PDFs, tables and scans, where most retrieval actually failsMost retrieval failures are ingestion failures — interleaved columns, repeated headers, tables flattened into rows that have lost their column names, OCR noise and chunks cut from their headings — and the first diagnostic is always to read twenty chunks with your own eyes.
- 41Permissions, freshness and metadata: the boring half that leaks dataAn index is a copy of every document with its access controls stripped off, so permissions must travel with each chunk and be enforced by your query code — never by an instruction to the model — and deletions must reach every index and cache or the document lives on.
- 42Long context instead of retrieval: the arithmetic of skipping the pipelineA million-token window can replace a retrieval pipeline for a small, stable corpus, and whether it should is decided by arithmetic — tokens per query times queries per day, with and without prompt caching — plus a measured check of multi-fact accuracy, not by the size of the window.
Module 5
Conversation, memory and the interface
The single call from the first module has no memory and no screen. This block builds both: where a conversation's state lives and how it is trimmed, how to condense a long history without losing the fact that mattered, what to keep across sessions and what a user must be able to see and delete, when to ask a question instead of guessing, the parts of a chat interface that are load-bearing, voice in and out within a latency budget, AI placed inside an existing screen rather than a chat box, the pipeline behind a file upload, and what the feedback buttons actually measure.
By the end you can
Build the stateful, user-facing half of an AI feature — a conversation store with a trimming and compaction policy, cross-session memory a user can inspect and delete, clarifying questions only where they pay for themselves, a chat or inline interface whose controls match what the model can and cannot do, a voice path with a stated latency budget, an upload pipeline with limits, and feedback capture whose numbers you can interpret rather than merely collect
- 43Where the conversation livesBecause the API forgets everything between calls, the conversation is your data structure, and the client that sends it is an untrusted party: store history server-side, trim by token budget without ever splitting a tool call from its result, and let nothing in the history carry authority.
- 44Compacting history: sliding windows, summaries and what gets lostA summary of dropped turns keeps a long conversation coherent but loses specifics by its nature — numbers, exact wording, negations — so hard constraints and facts the user stated must be extracted into a separate list that is appended and never summarised.
- 45Memory across sessions, and what a user must be able to seeCross-session memory turns conversations into a profile, so it must be extracted only from the user's own words and never from documents or tool results, restricted to categories you chose in advance, and visible and deletable by the user — because a remembered instruction is a persistent injection and a remembered inference is personal data nobody consented to.
- 46Asking instead of guessing: clarifying questions, slot filling and when a form beats a chatA model's default is to resolve ambiguity by guessing, so decide explicitly when a missing detail changes the action and cannot be safely defaulted — ask then, one question at a time, and otherwise state the assumption aloud — and recognise that fixed, structured input is a form's job, not a conversation's.
- 47The chat interface: the controls that are load-bearingThe text box is the trivial part of a chat interface; what matters is that every control tells the truth about the model — a stop button that cancels the upstream request, edits that branch the history, visible tool calls and sources, sanitised rendering of untrusted output, and an AI disclosure on the surface itself.
- 48Voice in and out: transcription, speech and the latency budgetA voice assistant is a cascade of endpointing, transcription, generation and speech whose delays add up, so a reply that feels natural comes from streaming every stage — partial transcripts, a model prompted for spoken register, and speech started on the first sentence — rather than from any one faster component.
- 49AI inside the existing screen: ghost text, drafts and the acceptance rateMost AI features belong inside the screen the user is already on — a proposed completion, a draft in the field, a summary at the top — where the model proposes and the person disposes, the cost is driven by call count rather than call size, and acceptance and edit distance replace thumbs as the measure.
- 50Handling uploads: the pipeline behind "attach a file"An attached file is an unvalidated request, an untrusted document and a piece of personal data at once, so it needs a type check by content, a route decided by size — into context or into retrieval — a delimited place in the prompt where its text is data, and a deletion that reaches every derived artefact, not just the original.
- 51Thumbs up, thumbs down, and what the buttons actually measureFeedback buttons are clicked on a few per cent of responses by people who are not representative, so treat them as a source of cases for review rather than a metric, prefer implicit signals that cover every response, and never optimise a model towards thumbs-up directly, because the shortest route to approval is agreement.
Module 6
Evaluating and iterating
A prompt change is a code change, and an earlier lesson said so. This block makes that practical for the person shipping the feature rather than the person studying measurement: the first fifty cases and where they come from, cheap assertions before any judge, a model judge you have checked against people, retrieval measured on its own, a harness that runs in CI without going red at random, traces complete enough to debug from, a weekly error-analysis habit, and a release that a number can stop.
By the end you can
Stand up an evaluation loop for a feature you own — a fifty-case set with provenance, deterministic assertions, a calibrated model judge with a stated agreement figure, a separate retrieval recall number, a CI harness with a threshold that tolerates a stochastic model, per-request traces, and a staged release with a stopping metric — and use it to decide whether a change ships
- 52Evaluating your own AI featureA prompt change is a code change; if you cannot re-run fifty saved cases, you are guessing.
- 53The first fifty cases, and where they come fromAn evaluation set for a feature is fifty cases with provenance — drawn from logs, from failures, and from the specification in roughly equal parts — each with an expectation precise enough to check by code or rubric, kept in version control beside the prompt it tests.
- 54Assertions before judges: the checks that cost nothingMost regressions in an AI feature are caught by deterministic checks — schema validity, forbidden strings, resolvable citations, length, language, code that runs, cost and latency bounds — which cost nothing per run and fail for a stated reason, so they go underneath any model judge, never instead of one.
- 55A model judge you have checked against peopleA model can grade your feature's answers at scale only once you have measured its agreement with human labels on a sample, written the rubric as binary criteria rather than a score, swapped the order to expose position bias, and used a different model from the one being judged.
- 56Measuring retrieval on its ownIn a system that retrieves and then generates, a wrong answer has two possible causes, and only a gold set of question-to-chunk pairs with a recall number tells you which — so build that set of forty questions in an afternoon and report retrieval and generation failures separately.
- 57The harness in CI, and why temperature zero is not deterministicA model at temperature zero still returns different tokens on different runs, because GPU arithmetic under batching is not bit-for-bit repeatable and providers change models under the same name, so a CI harness must judge by pass rate over repeated runs with cached responses for unchanged cases — or it goes red at random and gets switched off.
- 58Traces you can debug fromA wrong answer you cannot reproduce is diagnosable only from a trace that recorded, at the time, the prompt version, the retrieved ids and scores, every tool call with its arguments and result, and the usage and latency per stage — and because that trace is a copy of the user's words, it needs a retention rule and a deletion path of its own.
- 59Error analysis as a weekly habitOne person reading fifty failures a week and labelling each with a root cause from a short fixed list produces the only number that tells you which stage to work on next, turns every failure into a regression case, and keeps a team from fixing examples when the fault is a class.
- 60Shipping a change that can be stoppedA prompt or model change goes out behind a flag, is run in shadow against real traffic first, then reaches a sticky canary slice with a stopping metric decided in advance — because the evaluation set catches small regressions and the canary catches disasters, and neither does the other's job.
Module 7
Safety, security and the adversary
The failure-modes lesson said that text you did not write is data. This block is what follows from taking that seriously once real people, some of them hostile, are on the other end: how injection actually works and why filters for it fail, the channels through which a model leaks what it was shown, model output as untrusted input to everything downstream, moderation before and after the model, what you send to a provider and what you can verify about it, attacks on your bill, reducing hallucination by mechanism rather than instruction, attacking your own feature before someone else does, and the disclosures the law is beginning to require.
By the end you can
Threat-model an AI feature and defend it by architecture — name every channel by which untrusted text reaches the model and every channel by which its output reaches a privileged action or the outside world, restrict each with a control that does not depend on the model's obedience, layer moderation and abstention with measured false-positive costs, and state plainly what a user must be told and what you cannot promise about a provider's handling of their data
- 61The failure modes to design forText you did not write is data; it must never be able to trigger a privileged action.
- 62How prompt injection actually works, and why filters for it failA language model has no privilege separation in its input — system prompt, user message and a retrieved web page are all tokens of equal standing — so an instruction in untrusted text competes with yours on merit, pattern filters fail by paraphrase, and the only defences that hold are limits on what the model can do and a human between it and anything that matters.
- 63The channels that leak: how a model sends data outA model leaks what it was shown through whatever exit exists — an image tag it writes, a link it offers, a tool it calls with attacker-shaped arguments, a record it stores — so the defence is to enumerate every exit and close or gate each one, because the trifecta of private data, untrusted content and an open exit cannot be made safe by filtering.
- 64Model output as untrusted input to everything downstreamWhatever the model writes — HTML, SQL, a shell command, a file path, code — is untrusted input to the system that consumes it, and the classic web defences apply unchanged: sanitise before rendering, parameterise and restrict before querying, sandbox before executing, and never let generated text choose a path or a privilege.
- 65Moderation before and after the modelA moderation layer is cheap classifiers on the input and the output, arranged so that rules and small models handle the clear cases and a large model handles only the grey zone, with false-positive rates measured on your own traffic — and with distress routed to support rather than to a block, because a refusal is the wrong answer to a person in trouble.
- 66What you send to the provider, and what you can actually verifyEvery request is a disclosure of your prompt, your users' words and your customers' documents to a third party whose retention and training terms you can read but not audit, so minimise what leaves — redact, pseudonymise, strip — and tell users only what you know to be true.
- 67Denial of wallet: attacks on your billA public endpoint that calls a paid model can be made to spend a thousand dollars an hour by one script, so the defences — authentication before the model, per-user token budgets claimed before the call, caps on input and output, a spend alert and a kill switch — are part of the feature, not an operational afterthought.
- 68Reducing hallucination by mechanism, not by instructionTelling a model not to hallucinate does nothing measurable because it cannot tell which of its outputs are unsupported, so the rate is lowered by mechanisms — better retrieval, abstention below a score, claims that must cite and are checked against their source, constrained output — and it goes down, not to zero, which decides where a person must sign.
- 69Red-teaming your own feature before someone else doesAn afternoon spent attacking your own feature through every input channel — with a fixed category list, a severity grid and free automated probes — finds the failures before users do, and the findings become harness cases; assume the system prompt is public, and fix the blast radius where a refusal cannot be made perfect.
- 70Disclosure, consent and the law as it standsThe law on AI features differs by country and is still being written, but a set of minimums is safe everywhere — say it is AI on the surface, get specific consent for data use, offer a route to a person, make deletion real, keep records — while liability for wrong answers and copyright of outputs remain unresolved and must be presented as such.
Module 8
Cost, latency and running it in production
The first module priced a single call. This block prices a feature and keeps it running: the full cost model per thousand users, where the milliseconds go and which ones users feel, caching at three levels, the batch window at half price, routing between models by difficulty, the break-even arithmetic of running your own inference, rate limits in both directions, what breaks when a provider goes down or you switch, model deprecations and the pinning problem, and one feature walked from an empty repository to launch with every number filled in.
By the end you can
Produce a cost and latency budget for an AI feature at a stated scale, defend each line of it with a measurement, and operate the feature through a provider outage, a model deprecation and a tenfold traffic increase without an unplanned bill or an unplanned outage
- 71The cost model of a feature, line by lineA feature's cost is calls per user per day times tokens per call, split into cached input, uncached input and output at their three different prices, plus the fixed costs of indexing and evaluation — and once written as a spreadsheet, one variable usually dominates and that is the one to engineer.
- 72Where the milliseconds goA user feels time to first token and total time, and each is a sum of stages you can measure from traces — network, retrieval, reranking, the model's prefill, its generation, and a full extra round trip for every tool call — so the cheapest wins are prefix caching, a smaller model for the first token, parallel tools and streaming, not a faster provider.
- 73Caching at three levels: exact, prefix and semanticThree caches do three different jobs — an exact-match response cache for repeated requests, the provider's prefix cache for repeated context, and a semantic cache that serves similar questions and is the one that can return a wrong answer — and every cache must be keyed on prompt version, model and corpus version, and must never serve one user's content to another.
- 74The batch window: half price for anything that can waitProviders sell a batch endpoint at around half the interactive price in exchange for results within a day rather than seconds, so every model call that is not a person waiting — classification, backfills, evaluation runs, embedding, summaries — belongs in a queue that drains through it, with idempotent items and per-item failure handling.
- 75Routing and cascades: paying the high price only where it is neededMost requests are easy and a cheap model handles them, so a router that sends easy traffic to the small model and escalates on a measurable signal — a failed validation, a refusal, low judge confidence, a hard-intent label — cuts cost by more than half, provided the escalation rule actually fires and each route is evaluated on its own slice.
- 76Running your own inference: the break-even arithmeticSelf-hosted inference is priced by the GPU hour, not the token, so its cost per million tokens is the hourly price divided by throughput times utilisation — and at the bursty, daytime utilisation most products have, a rented card costs several times the per-token price of the same open model from an API, unless data, capability or sustained load forces the choice.
- 77Rate limits in both directionsThe provider limits your requests and tokens per minute by tier and raises limits only on request over days, so a feature must be planned at half its allowance to survive bursts, must throttle itself client-side rather than discover the limit through 429s, and must set its own ceiling for users below the provider's so that refusals are yours, clear and recoverable.
- 78Outages, fallbacks and what does not portA fallback to a second provider must be tested by deliberately breaking the primary, because tool-call formats, structured-output modes, tokenisers, stop reasons and prompt behaviour all differ beneath any abstraction layer — and because falling back also sends user data to another company, the privacy notice must already name it.
- 79Deprecations and the pinning problemAn undated model alias changes behaviour silently whenever the provider ships a new version, and dated snapshots retire on a published schedule, so production pins a snapshot, a quarterly check reads the deprecation page, and every upgrade goes through the harness, shadow and canary before the provider's date forces it.
- 80One feature, end to end, with every number filled inA support-triage assistant for a small company, built with every lesson in this course, costs about a hundred dollars a month, drafts a reply before the agent opens the ticket, and is bounded against injection not by any prompt but by having no tool that can act — and the walk-through shows which lessons were used and which features were deliberately left out.