Intermediate to Advanced AI

The list is right about the parts and silent about the assembly

Read the source list again and notice what it does. It names good things. Karpathy's videos really are the best mid-to-advanced LLM content available for free. The Hugging Face courses really are excellent and hands-on. Reading papers really is the advanced curriculum. Every individual judgement is defensible.

Then look at its closing recommendation: Karpathy's videos → Hugging Face NLP/agents course → pick one paper a week and reimplement it. That is three items, and the arrow between the second and third is doing an enormous amount of unearned work. There is a chasm between "I completed a course where the fine-tuning script was written for me" and "I can read a paper cold and rebuild its central claim." The list gestures across that chasm with an arrow.

The dependency graph

The order in this course is not a taste preference. It falls out of what genuinely depends on what. Four dependencies do most of the work:

• You cannot reason about training without the mechanism. Every training decision — learning rate, batch size, what a loss spike means — is a statement about gradients flowing through a specific architecture. Without the forward and backward pass in your hands, training advice is folklore you are memorising.
• You cannot post-train without training. SFT, DPO and GRPO are all gradient descent on a modified objective. If the base loop is a black box, the modification is a black box wrapped in an acronym.
• You cannot evaluate what you cannot change. Evals are only useful when they are attached to a lever. Learning evals before you can move a model produces someone who writes excellent dashboards for systems they cannot improve.
• You cannot optimise inference before you know what the model does at inference. KV caching, batching and speculative decoding are all consequences of one fact about the decode loop, and that fact is invisible until you have written the loop.

Which gives the spine: mechanism → training → post-training → evals → inference. Evals sit fourth rather than last, which surprises people, because by that point you have a model you can actually change — and because everything after it is optimisation, and you should not optimise a system you cannot measure.

What "advanced" actually means here

Worth defining, because the word is doing unexamined work in the source list. Three candidate definitions circulate, and only one of them is a curriculum:

This course uses the third. That is why every stage ends with a clearing test phrased as "you can do X unaided" rather than "you completed module Y." The difference is not rhetorical. Finishing a course is something that happens to you; doing a thing unaided is something you do.

Scan this first. It is the compressed version of the module, and the rows that need explaining get it below.

Still excellent — use these today

The shape of the gap

Look at the list's implicit model of the job. It has fundamentals (fast.ai, Stanford), it has building with LLMs (Hugging Face, Karpathy), it has production-adjacent (FSDL), and it has papers. That is a 2023 map of the work. It is missing the three things that consume most of the effort in an AI organisation in 2026, and they are missing for a structural reason: they are the parts that only become visible after you have something in production. Course-shaped content lags them, because courses are written by people teaching the building, and these three are about the keeping-it-working.

Gap 1 — Evals

The list mentions evals exactly once, inside the FSDL entry — that is, inside the frozen 2023 resource, whose evals material predates current practice. So in effect the gap is total.

This is the most consequential omission because evals are the bottleneck discipline. Nearly every organisation that fails to ship an AI feature fails at the point where someone asks "is it good enough?" and there is no defensible answer. Module 7 teaches the discipline. The resources, all checked:

• AI Evals for Engineers & PMs — Hamel Husain and Shreya Shankar. Paid, cohort-based on Maven, and the material was completely refreshed for the September 2026 cohort. They have taken 2,000+ engineers and PMs through it, including teams at OpenAI and Anthropic. It is the closest thing the field has to a canonical evals curriculum.
• Hamel's evals FAQ and blog — free, and covers most of the intellectual content of the course without the cohort. Start here before you pay for anything.
• Inspect — the UK AI Security Institute's open-source eval framework, MIT-licensed. It has become the de facto standard for frontier-grade evaluation, with one interface across all major providers plus local vLLM/Ollama, and sandboxing for agentic evals that execute untrusted code. The companion inspect_evals repo is a large community collection across coding, maths, cybersecurity and reasoning.

Gap 2 — Inference optimisation

The list mentions vLLM and llama.cpp once, in passing, as repos worth rebuilding. It does not say what you would learn or in what order, and it treats inference as an implementation detail rather than a discipline.

It is not a detail. For any product at volume, inference cost is the dominant line item and the thing that decides whether the feature has a business. It is also where the most transferable systems knowledge in modern ML lives, because the constraints are legible: memory bandwidth, cache size, batch scheduling. Module 8 teaches it. Resources:

• CS336 Lecture 10 (Inference) — free on YouTube, and the best single hour on the subject at the right level of abstraction.
• GPU MODE — over 100 lectures with code, actively maintained, covering CUDA fundamentals, memory architecture and profiling, Triton, FlashAttention, Ring Attention, quantization, NCCL multi-GPU communication, speculative decoding and kernel fusion. This is the deep end and it is free. There is an active Discord and a YouTube channel.
• vLLM docs — read them as a text on serving architecture, not as an API reference. The PagedAttention design is the clearest available worked example of applying an OS idea to an ML problem.

Gap 3 — Post-training

The list gets closest here — it names RLHF inside the Hugging Face entry. But it treats post-training as one item among several rather than as the stage where most of the observable quality of a modern model is created. Everything you experience as a model's helpfulness, format discipline, refusal behaviour and reasoning ability is post-training.

• The RLHF Book — Nathan Lambert. Free web version, print edition published by Manning in July 2026. It covers instruction tuning, reward modelling, rejection sampling, RL, direct alignment algorithms and on-policy distillation. This is the reference text for Module 6.
• HF smol course — the hands-on complement. Instruction tuning, evaluation, preference alignment, vision-language models, with RL and synthetic-data units rolling out.
• Build a Reasoning Model (From Scratch) — Sebastian Raschka, out in 2026. Starts from a pretrained model and works through evaluation, inference-time scaling, RL and distillation on a small Qwen3 base. The natural successor to his Build a Large Language Model (From Scratch), which passed 100,000 GitHub stars.

The biggest omission is a whole course

One more, for the exec reading

Chip Huyen's AI Engineering (O'Reilly, December 2024) is not on the list and should be. It is the one book that treats foundation-model applications as an engineering discipline with a full chapter each on evaluation methodology and inference optimisation — two of the three gaps. It will not make you able to build any of it, which is why it is not a substitute for Stages 1–5. It will make you able to ask the right question in a design review, which is a different and also useful thing.

Attention is a soft dictionary lookup

Start with something you already know. A Python dictionary takes a key, finds the one entry whose key matches exactly, and returns its value. Hard match, one winner, no gradient.

Attention is the same operation made differentiable. Every position in the sequence emits three vectors. The query is what this position is looking for. The key is what this position offers as a label. The value is what this position will hand over if selected. To compute the output at position i, you take its query, dot it against every key in the sequence to get a similarity score, softmax those scores into weights that sum to one, and return the weighted average of all the values.

So instead of "find the one matching entry," you get "return a blend of all entries, weighted by how well each matches." Hard lookup becomes soft lookup. That single change is what makes it trainable: gradients flow back through the softmax to every position, so the model can learn what to look for and what to advertise.

The scaling factor — dividing by the square root of the head dimension before the softmax — exists for a boring and important reason. Dot products of high-dimensional vectors have variance proportional to the dimension. Without the scale, the scores get large, the softmax saturates into a near-one-hot distribution, and the gradient vanishes. It is not a theoretical nicety; it is the difference between a model that trains and one that does not.

The residual stream — the idea most explanations skip

Almost every diagram of a transformer draws it as a stack: input goes in the bottom, gets transformed by block 1, that output gets transformed by block 2, and so on. That picture is wrong and it will mislead you.

Because of the residual connections, each block does not transform its input. It reads its input, computes something, and adds the result back. Written plainly, a block is:

# not: x = block(x)  — a pipeline of transformations
x = x + attention(layernorm(x))   # read, compute, add back
x = x + mlp(layernorm(x))         # read, compute, add back

So there is a single vector per position — call it the residual stream — that runs unbroken from the embedding all the way to the output head. Every block is a side-channel that reads from the stream and writes an increment back into it. Nothing ever replaces the stream; things are only ever added to it.

Three things follow immediately, and each one is a question people otherwise find mysterious:

• Why can you delete a middle layer and get graceful degradation? Because you removed one contribution from a sum, not a stage from a pipeline. Deleting stage 7 of a pipeline breaks everything downstream. Deleting one addend from a sum of forty makes the sum slightly wrong.
• Why does activation steering work? Because if you can find a direction in the residual stream that corresponds to a concept, adding a multiple of that vector is exactly the same kind of operation the model's own blocks perform. You are not hacking it; you are writing to the stream in its native format.
• Why does depth buy refinement rather than re-representation? Because later layers see an accumulated draft rather than a foreign encoding. The model is doing successive editing, not successive translation.

If you take one idea from Stage 1, take this one. It is the load-bearing intuition for interpretability, for fine-tuning behaviour, and for why LoRA works at all.

Attention routes; the MLP knows things

The last piece of the mechanism is the division of labour, and it is cleaner than people expect.

Say it as one sentence: attention moves information between positions; the MLP transforms information at a position. Once you hold that, a lot of empirical results stop being surprising — that factual recall localises to MLP layers, that mixture-of-experts routes MLPs rather than attention heads, and that context-length work is almost entirely attention work.

And the training objective is embarrassingly simple

Worth stating plainly because it is easy to over-mystify. The pretraining loss is next-token cross-entropy and nothing else. Show the model a prefix, ask for a distribution over the next token, penalise it by the negative log probability it assigned to the token that actually came next. Average over every position in every document.

There is no auxiliary objective for reasoning, no term for truthfulness, no supervision signal for helpfulness. Everything that later looks like capability is a side effect of compressing a very large corpus under that one objective. Holding this clearly is what stops you from believing the model was taught anything in pretraining, and it is what makes Stage 3 comprehensible — post-training exists precisely because next-token prediction gives you a document completer, not an assistant.

What to actually watch and read

Scaling laws are a unit-economics argument in disguise

The Chinchilla result is usually taught as a fact about model quality. It is more useful taught as a fact about budgets.

Start with the approximation that makes the whole thing tractable: training a dense transformer costs roughly C ≈ 6ND FLOPs, where N is parameters and D is training tokens. (The 6 is two for the forward pass and four for the backward, per parameter per token — approximately, and the approximation is good enough to plan with.)

Now fix C, because in practice you fix C — you have a cluster for a month. You are choosing a point on a hyperbola: a big model on few tokens, or a small model on many. Both cost the same. The empirical finding is that loss is minimised near the middle, and specifically at roughly 20 tokens per parameter. Before that result, the field systematically built models that were too large and trained them on too little data.

Arithmetic intensity, or: why your GPU is mostly waiting

This is the concept that makes systems work click, and it is a ratio.

A modern accelerator can do on the order of a thousand trillion floating-point operations per second. It can read from its high-bandwidth memory at a few terabytes per second. Divide those and you get an awkward number: the chip can perform hundreds of arithmetic operations in the time it takes to fetch a single number from memory.

So for any kernel, the question that decides its speed is how many FLOPs do you do per byte you read? That ratio is arithmetic intensity. High intensity — a big dense matrix multiply, where each loaded value participates in many multiply-accumulates — and you are compute-bound, which is the good case: the expensive silicon is busy. Low intensity — an elementwise operation, where you read a number, do one thing to it, write it back — and you are memory-bandwidth-bound, and the FLOPs sit idle no matter how many you paid for.

The four parallelisms, and what each one costs

When one device is not enough, you split. There are four axes and confusing them is a common source of nonsense in design discussions:

Real systems compose them — tensor parallel within a node where the interconnect is fast, pipeline parallel across nodes where it is not, data parallel over the whole thing. The rule of thumb worth memorising: tensor parallelism inside a node, pipeline parallelism between nodes, data parallelism everywhere.

What to actually do

This is the stage with a real compute bill, and it is small. The centrepiece is nanochat: tokenizer, pretraining, midtraining, SFT, evaluation, inference and a web UI, in one legible repository. The README currently claims a GPT-2-capability model for about $48 of 8×H100 time — roughly two hours — against something like $43,000 for the equivalent in 2019. Run the cheap tier once, end to end, and you will have done something most people talking about this have not.

The problem post-training solves

A pretrained model is a document completer. Ask it "What is the capital of France?" and a faithful next-token predictor might reasonably continue with "What is the capital of Germany?" — because in its training corpus, that string most often appeared in a list of quiz questions. It is not confused. It is doing exactly what it was optimised to do.

Post-training is everything that turns that object into something that answers. It is conventionally three moves, and it is worth being precise about what each one can and cannot do.

RLHF classic, and then the simplification

The original recipe has three steps. Collect human comparisons of model outputs. Train a reward model — a network that takes a response and returns a scalar — to predict which one a human would prefer. Then use reinforcement learning, usually PPO, to update the policy to maximise that predicted reward, with a KL penalty against the original model so it does not drift into gibberish that happens to score well.

It works and it is a nuisance: three models in memory at once (policy, reward, reference), an RL loop that needs online sampling, and enough hyperparameter sensitivity that reproducing someone else's result is a project.

The change that actually mattered: reward stopped being taste

The most important shift in post-training between 2023 and 2026 is not an algorithm. It is where the reward signal comes from.

That is what unlocked the reasoning-model generation. Once the reward is a verifier rather than a critic, you can afford enormous numbers of rollouts, and the model discovers behaviours — checking its work, backtracking, trying a second approach — that nobody demonstrated, because those behaviours raise the probability of passing the check.

GRPO is the algorithmic simplification that goes with it. PPO needs a learned value network to estimate the baseline for the advantage. GRPO drops it: sample a group of completions for the same prompt, score them all with the verifier, and use the group's mean as the baseline. Advantage becomes "how much better than my siblings was this attempt." One fewer network, and it suits the setting exactly, because generating a group of samples for one prompt is cheap when you are already sampling.

The strategic read, which is the part worth carrying into a product conversation: a capability is amenable to RL exactly to the degree that you can write a checker for it. That is a sharper predictor of what models will get good at next than any parameter count. Domains with cheap verifiers race ahead; domains where quality is a matter of judgement move at the pace of preference data.

What to actually do

A benchmark is not an eval

These get used interchangeably and they answer different questions.

An eval is also not a model property. It measures the system — prompt, retrieval, tools, parsing, fallbacks, and the model. Swapping to a better-benchmarked model routinely makes a system worse, because the prompt was fitted to the old one. If your eval cannot detect that, it is measuring the wrong object.

The workflow that works, in the order it works in

Most teams start by choosing metrics. That is the error. The sequence that actually produces improvement inverts it:

1. Look at data. Read 100 real traces end to end. Not summaries, not aggregates — the actual conversations, with what the user asked and what the system did.
2. Open-code the failures. For every trace that went wrong, write one plain sentence describing what went wrong. Do not categorise yet. Do not reach for a taxonomy someone else wrote. Just describe, in your own words, one line per failure.
3. Axial-code into a taxonomy. Now group your sentences. The categories emerge from your data rather than from a blog post, which is the whole point — you will find failure modes specific to your domain that no generic rubric contains.
4. Count. Frequency per category. This is usually the moment the project changes direction, because the thing everyone was arguing about turns out to be 3% of failures and the thing nobody mentioned is 40%.
5. Build a targeted eval for the biggest bucket. Now, and only now, do you write a metric. It measures one specific failure mode you have evidence matters.
6. Fix, re-measure, repeat.

LLM-as-judge is a classifier you have not validated

Using a model to grade outputs is the only way to evaluate open-ended generation at volume, and it is fine. What is not fine is treating the judge's output as ground truth. A judge is a classifier. You would not ship an unvalidated classifier, and this is one.

Validate it the way you would validate any classifier, with one wrinkle that matters:

• Label a stratified sample yourself. Not your intern — you, or whoever owns the product definition of "good." A few hundred examples, deliberately including hard and borderline cases rather than a uniform random draw.
• Measure both error rates separately. Not "agreement" — that single number hides everything. Measure the true positive rate (of the outputs you called good, how many does the judge call good) and the true negative rate (of the outputs you called bad, how many does the judge catch) as two separate numbers. Judges are systematically lenient: they pass things they should fail, which shows up catastrophically in TNR and barely at all in overall agreement.
• Correct your headline number using those rates. If your judge passes 80% of traffic, but you measured its TPR at 0.95 and TNR at 0.70, the raw 80% is not your success rate. You can back out a corrected estimate from the observed pass rate and the two measured rates — and the corrected number is the one you take to a review.
• Re-validate when anything changes. New judge model, new prompt, drifted traffic — your rates are stale and so is every number computed from them.

What to actually do

One fact explains almost everything

Generation has two phases, and they could hardly be more different.

Sit with the decode case, because it is the one that costs money. A 70-billion-parameter model in 16-bit is 140 GB of weights. Every generated token requires reading all 140 GB. At a few terabytes per second, that is a hard floor of tens of milliseconds per token per sequence, and it is set by memory bandwidth, not by arithmetic. You could double the chip's FLOPs and change nothing.

So: nearly every inference optimisation is an attempt to get more useful work out of each byte read from memory. That sentence organises the rest of the module, and once you have it, you can usually derive the technique rather than memorise it.

The four levers, all of them the same lever

1. Batching — amortise the read

If reading the weights once produces one token, that read costs you a token. If you have 64 sequences in flight and read the weights once to advance all 64, the same read produces 64 tokens. You have divided the bandwidth cost per token by 64, for free, because the FLOPs were idle anyway. This is why batching is the first and largest lever, and why a serving system with low batch occupancy is burning money in a way that no amount of model optimisation will fix.

Continuous batching is the refinement that matters. Naive batching runs a batch to completion, so a batch of 32 in which 31 sequences finish early runs at 1/32 utilisation until the straggler finishes. Continuous batching schedules at token granularity: as soon as a sequence emits its stop token, its slot is freed and a queued request takes it on the very next step. Same hardware, several times the throughput.

2. The KV cache — the constraint you did not budget for

To avoid recomputing attention over the whole prefix at every step, you cache the keys and values for every previous token. That cache is what makes decoding tractable. It is also, in production, usually the thing that runs out.

It grows with batch size × sequence length × layers × heads, it lives in the same HBM as the weights, and at long contexts and high batch sizes it can exceed the model itself. Your maximum concurrency is therefore usually set by KV cache capacity, not by weights and not by compute.

3. Speculative decoding — buy k tokens for one read

A small draft model proposes the next k tokens cheaply. The large model then verifies all k in a single forward pass, because verification is a prefill-shaped operation — you are scoring a known sequence, not generating one. Accepted prefixes are kept; the first rejection resets. You have paid one big-model memory read for up to k tokens instead of k reads. Another bandwidth trade, and the acceptance rate is what determines whether it pays.

4. Quantization — smaller bytes, same count

Moving weights from 16-bit to 8-bit halves the bytes you read per token, so it roughly halves your decode bandwidth cost. Note the consequence people get wrong: quantization helps decode far more than prefill, because prefill was compute-bound and you did not shrink the FLOPs. If someone reports a 2× speedup from quantization on a prefill-heavy workload, ask what else changed. FP8 has become the practical default in serving stacks, and the largest independent study (Llama-3.1 family, 500,000+ evaluations) reports it as effectively lossless. The throughput claim needs more care: halving the bytes halves the weight-read term, but the KV cache does not shrink unless you quantize it too, and prefill does not shrink at all. Doubling is the best case, not the typical one — Stage 5 separates the three cases.

The exec version

What to actually do

What "reimplemented it" has to mean

The source list's best advice is "pick one paper a week and reimplement it," and it never says what that means. Without a definition, the word collapses into whatever you did, which is how people end up with eight repos and no capability.

Three properties make that definition work. It is falsifiable — either your number matches the trend or it does not. It is affordable — "at a scale you can afford" means a 10-million-parameter model reproducing a shape, not a 70-billion-parameter model reproducing a value. And it includes the diff step, which is where most of the learning actually happens, because the gap between your version and theirs is a list of things you did not know mattered.

The ladder

Ordered by dependency, not by publication date or by fame. Each rung assumes the ones before it. Hours assume you have cleared the relevant stage.

Total: roughly 144 hours. At five hours a week that is about seven months — which is the honest number, and it is why "one paper a week" was never going to survive contact with a calendar. Eight papers properly reimplemented over seven months will take you further than fifty papers read.

If you are reading this partly as a product person rather than only as a learner, this module is for that. The path above is one learner's plan. The interesting question is what happens if you try to run it for fifty thousand people.

The gradeability gradient

The clearing tests in Modules 4–8 were written to be artefact-producing, which was a pedagogical choice. It is also a product choice, because artefacts are gradeable and feelings are not. But they are not equally gradeable, and the gradient is the design constraint:

So four of five stages are largely machine-gradeable, and the fifth — evals — is the one where a cohort earns its price. That is a clean product decomposition: self-paced with automated grading for 1, 2, 3 and 5; cohort-based with human review for 4. It also happens to be the honest one, since evals is the stage where the skill genuinely is taste developed against feedback.

The compute line, which is smaller than people assume

The obvious objection to a course like this is that it needs GPUs and therefore cannot scale economically. The numbers say otherwise, and the change is recent:

A few hundred dollars of compute per learner for a course that produces a demonstrably capable engineer is not an unusual cost structure — it is comparable to a lab science course, and far below a bootcamp. The thing that changed is the denominator: the same capstone cost roughly a thousand times more seven years ago. A course that was economically impossible in 2019 is now merely expensive, and that shift has not yet been absorbed by most curriculum planning.

The second-order design question is who holds the budget. Learner-pays makes failure feel expensive, which is the wrong incentive on a path where failed runs are the curriculum. Platform-pays with a per-learner cap is better pedagogy and a harder P&L conversation. Worth having the conversation with the number in hand rather than the vibe.

Why nobody ships the sequence

Three things that transfer to any curriculum

• Date every resource, visibly. Half of this course's value came from checking last-modified dates that took two minutes each. A catalogue that surfaces "last refreshed" next to every item would prevent most of what Module 2 documents — and note that it is a metadata problem, not a content problem, which makes it unusually cheap to fix.
• Completion is the wrong unit; capability is the unit. "You can do X unaided" is harder to instrument than "watched 87% of the video," and it is the only claim a learner can take to an employer. The clearing tests here are all artefact-producing precisely so that the claim is checkable by someone who does not trust the platform.
• Sequence is the product; content is the commodity. Everything in Modules 4–8 points at free external material. The value added is the ordering, the dependency reasoning, the honest time budget and the tests. That is a defensible position and it does not require owning any of the underlying content — which is a different business than most catalogue thinking assumes.

The tool asks you to commit to an entry stage before it computes one. That ordering is deliberate. If you see the answer first, you will find reasons to agree with it, and the most useful output of this exercise is the case where your guess and your answers disagree.