Build RAG/tools/agent workflows with evals and a production feedback loop
Locked
Scored stage progress
0%
Practice and implementation
0/20
Exercises + milestone requirements owned by this stage
Evidence and gate
0 · 0/13
Artifacts · checked requirements
Why it matters
Purpose of this stage
This is where LLM APIs become useful. Do not enter this stage as a substitute for previous foundations.
Subjects
6 subjects · 54 canonical topics
Each subject states what you should be able to do, an order to work through it in, what to watch for, and the questions a reviewer should be able to ask you at the end. None of it is tracked or scored.
How to work through this subject
An order that works, not a checklist. Nothing here is tracked — the steps sequence this subject’s own topics and add none.
Write the prompt as a function input: named values in, one defined output shape out.
Define the output schema and validate every response against it.
Decide what happens when the response does not match — repair, retry, or fail — and implement it.
Add a retry policy with a limit and a different behaviour on the last attempt.
Wire up one tool call and define exactly what arguments the model may supply.
Measure your context budget and decide, in advance, what is dropped first when it is exceeded.
Finish by naming which parts of the system must be deterministic and showing how you ensured it.
What to watch for
Structured output is what makes a model call testable. Free text is not an interface.
Every model call is stochastic by default. Decide deliberately where you need determinism.
A retry that sends the identical request and hopes is not a strategy. Change something, or stop.
The context budget is finite and shared between instructions, retrieved content and history. Something is always being dropped; choose what.
A tool call is your code executing on the model's suggestion. Validate the arguments as you would any untrusted input.
Curriculum topics
prompts as program inputs
Treating the prompt as a function argument with a defined contract, not as conversation.
structured outputs
Requiring the model to answer in a defined shape, such as JSON, so the result is machine-checkable.
schemas
The declared shape a response must satisfy, and the thing you validate it against.
tool/function calling
Letting the model request that your code run a named function with arguments, rather than answering directly.
retries
Re-attempting a failed or malformed model call, with a limit and a plan for when it keeps failing.
deterministic vs stochastic behavior
The same prompt may not give the same answer twice. Decide where your system requires determinism.
context budgeting
Deciding what fits in the context window and what gets dropped first when it does not.
Resources for this subject
REFERENCEPrompt Engineering Guide DAIR.AI Techniques catalogue for treating prompts as program inputs. Use the structured-output and tool-calling sections; skip the certification-adjacent material.Open
A reviewer should be able to answer these
Answer from memory, out loud, before moving on. These are not tracked and do not affect the gate — they tell you whether you actually understand this subject.
What does requiring structured output give you that free text does not?
How do you handle a response that does not match your schema?
What should happen on the second and third retry, and when do you stop?
What is in your context budget, and what gets dropped first when it is exceeded?
Which parts of your system must be deterministic, and how do you ensure that?
What makes a prompt a program input rather than a message?
How do you validate the arguments a model supplies to a tool?
Exercises
Demonstrated practice · 35% base weight
This stage defines implementation through its owned project milestone rather than a separate exercise list.
Resource status is a convenience marker only. It cannot change competencies, gates, or stage completion — in any tier below.
References
1 available
OPTIONALHugging Face Agents Course Hugging FaceAn optional course on tool use and agent loops. Take it only after your deterministic workflow works and you can show why agency was required — 24_SCOPE_CONTROL.md is explicit that frameworks come after the fundamentals.Only after LLM + evaluation foundations.Open
Need another explanation?8 supplementary resources · not part of the canonical library
Use these for a demonstrated gap, not as a parallel course. They are authored additions and can never be the primary resource for a subject.
REFERENCEPrompt Engineering Guide DAIR.AITechniques catalogue for treating prompts as program inputs. Use the structured-output and tool-calling sections; skip the certification-adjacent material.Open
OPTIONALRetrieval-Augmented Generation for Knowledge-Intensive NLP Tasks Lewis et al., 2020The paper that named RAG. Worth reading to see how modest the original claim was compared to current usage.Open
REFERENCEHybrid Search Explained WeaviateHow lexical and vector retrieval combine, and why the hybrid usually wins. Supports the stage's hybrid/reranking experiment.Open
OPTIONALChroma documentation ChromaOne small vector store, sufficient for the stage project. Scope control bars vector databases before Stage 8 and warns against collecting five of them.Deliberately the only vector store listed. Learn retrieval metrics before adopting infrastructure.Open
OPTIONALRagas Exploding GradientsMetrics for evaluating retrieval and generation separately. Build your own eval set first — the stage requires that before optimisation.Open
REFERENCEOpenAI — Evaluating model performance OpenAIThe current official documentation for defining eval datasets and graders, and for running an eval as a repeatable job. Read it for the vocabulary — datasets, graders, runs — then implement the same shape yourself.Provider-specific documentation for a provider-independent idea. The concepts this stage requires — eval sets, graders, component vs end-to-end evals, regression evals — must be implementable without any vendor.Open
REFERENCECreate strong empirical evaluations AnthropicHow to design task-specific evals, choose between code, human and model graders, and where LLM-as-judge stops being trustworthy. Vendor-neutral in substance.Open
REFERENCEOpenTelemetry — semantic conventions for generative AI OpenTelemetryA vendor-neutral field vocabulary for tracing LLM calls: prompts, tokens, latency, errors. Use it to shape your own trace schema instead of adopting a tracing product.Deliberately vendor-neutral. No LangSmith or hosted tracing platform is required at any stage.Open
Stage gate
Readiness assessment
Not ready
Stage-specific requirements
Answer each prompt without notes. Expand a question only when you are ready to check your answer.
0/5
Demonstrate with evidence that RAG/agent design improved the task instead of merely adding complexity
Answer
Show the baseline first, then the numbers. A defensible answer reports retrieval metrics and generation quality separately, on an evaluation set built before the change, and includes latency and cost. It names what got worse as well as what improved, and states what the added complexity bought that the simpler design could not deliver.
Convert a production failure into a reproducible eval case
Answer
Take the trace of the failure, extract the input and any context it depended on, and write a case that reproduces the wrong outcome today. Define the expected behaviour precisely enough to be checked automatically, add it to the versioned eval set, and confirm it fails before the fix and passes after it. If the trace does not contain enough to replay the failure, fix the trace schema first.
Run a regression eval that would catch that failure returning
Answer
A regression eval is the versioned set of cases, including converted failures, run on every change so a fixed problem cannot silently return. It reports per-component results rather than one score, so a regression is traceable to retrieval, generation or tool selection. It includes latency and cost, because a slower or more expensive system is also a regression.
State the task-success metric you are optimizing
Answer
Name the metric in the language of the task, not the model: the fraction of requests that produced the outcome the user wanted, however your domain defines that. Say who decides whether an instance succeeded, how that judgement is captured, and what the metric does not cover. If nobody can say whether a task succeeded, there is no metric yet.
Compare two approaches with evidence rather than preference
Answer
Run both on the same evaluation set, report quality, latency and cost side by side, and say which cases each one wins and loses. State what the difference means for the decision the system supports, and whether it is large enough to justify any extra complexity. Preference is not evidence, and a single aggregate score is not a comparison.
Universal requirements
The eight conditions 01_LEARNING_RULES.md applies to every stage. Expand one to see what it actually asks for.
0/8
Core concepts can be explained from memory.
What this asks for
Every subject in this stage has review questions. The condition is met when you can answer them out loud, without notes and without looking anything up, in your own words rather than a recited definition. An explanation you can only give with the material open is recognition, not recall.
Required exercises are complete.
What this asks for
Complete means each exercise satisfies its stated requirements and its done-when criteria, not that you attempted it. Where an exercise says you should be able to explain or defend something, that explanation is part of completing it. A stage whose source lists no exercises meets this through its owned project milestone instead.
Stage project works.
What this asks for
Works means it runs end to end from a clean checkout, on the inputs it was built for, with the failure cases it claims to handle actually handled. It is not a notebook that produced a result once. Someone else following your README should be able to run it without asking you a question.
Tests or verification exist.
What this asks for
There is something that would fail if the behaviour regressed, and you have watched it fail. That can be automated tests or a written verification procedure with recorded results, depending on what the stage produces. A test that has only ever passed is evidence of nothing.
README explains decisions.
What this asks for
The README says what the project is, how to install, run and test it, and why it is built the way it is: the choices you made, the alternatives you rejected, and what you deliberately left out. A list of commands with no reasoning does not meet this condition.
You can reproduce the core implementation without copying.
What this asks for
Delete the central piece and write it again from an empty file, without your previous version, a tutorial, or generated code in front of you. It does not have to be identical, it has to work and you have to be able to explain every line. This is the condition that separates having finished from being able.
You pass the stage self-assessment at >= 80%.
What this asks for
Record a gate exam and score yourself honestly against the dimensions the roadmap defines: recall, implementation, debugging, testing and design judgment. Elapsed time, hours watched and completed resources count for nothing here. Checking the boxes without a recorded exam does not open the gate.
Claude/ChatGPT reviewer finds no foundational blocker.
What this asks for
Ask an AI reviewer to examine the work against this stage's concepts and report foundational misunderstandings, not style. A foundational blocker is a misconception that later stages would build on, such as leakage, a broken abstraction, or an explanation that is confidently wrong. Nothing in that class may remain open when you pass.
Recorded exam
No exam recorded. Checking every requirement is not sufficient.