Data Analysis + Data Engineering Foundations

Load the file and report shape, column names, types and a sample of rows. Count missing values and duplicates per column, look at the extremes of every numeric column, and check that dates parsed as dates. State your schema assumptions explicitly, then say which of them the file actually violates.

Express each cleaning decision as a deterministic function that takes data and returns data, leaving the raw file untouched. Handle types, missing values, duplicates and malformed rows explicitly, recording how many rows each step affected. Write down why each decision was made, not only what it did, and keep raw and cleaned outputs in separate locations.

State the expected schema before looking at the data, then check the loaded file against it: columns present, types correct, nullability as declared. Add checks for values that are impossible rather than merely unusual, and for duplicate records where no natural identifier exists. Failures should be loud, specific and reported in full rather than one at a time.

Report the counts first: rows used, rows dropped, missing values per column. Then give descriptive statistics per numeric column and the distribution behind them, since a mean alone hides shape. Summarise the categorical columns by their value counts, and state the question each summary answers.

Say what the dataset represents, where it came from and when it was captured, then what you had to change to trust it and what you could not resolve. Name the columns you would not use and why, including any that could not have been known at the time the row was created. Finish with what the data does not support concluding.

Delete every intermediate and cleaned output, then run the pipeline again from the raw data alone and compare the result to the previous cleaned dataset. They should be identical. Anything that differs is a step depending on the clock, on random values, on manual edits, or on state left behind by the earlier run.

Every subject in this stage has review questions. The condition is met when you can answer them out loud, without notes and without looking anything up, in your own words rather than a recited definition. An explanation you can only give with the material open is recognition, not recall.

Complete means each exercise satisfies its stated requirements and its done-when criteria, not that you attempted it. Where an exercise says you should be able to explain or defend something, that explanation is part of completing it. A stage whose source lists no exercises meets this through its owned project milestone instead.

Works means it runs end to end from a clean checkout, on the inputs it was built for, with the failure cases it claims to handle actually handled. It is not a notebook that produced a result once. Someone else following your README should be able to run it without asking you a question.

There is something that would fail if the behaviour regressed, and you have watched it fail. That can be automated tests or a written verification procedure with recorded results, depending on what the stage produces. A test that has only ever passed is evidence of nothing.

The README says what the project is, how to install, run and test it, and why it is built the way it is: the choices you made, the alternatives you rejected, and what you deliberately left out. A list of commands with no reasoning does not meet this condition.

Delete the central piece and write it again from an empty file, without your previous version, a tutorial, or generated code in front of you. It does not have to be identical, it has to work and you have to be able to explain every line. This is the condition that separates having finished from being able.

Record a gate exam and score yourself honestly against the dimensions the roadmap defines: recall, implementation, debugging, testing and design judgment. Elapsed time, hours watched and completed resources count for nothing here. Checking the boxes without a recorded exam does not open the gate.

Ask an AI reviewer to examine the work against this stage's concepts and report foundational misunderstandings, not style. A foundational blocker is a misconception that later stages would build on, such as leakage, a broken abstraction, or an explanation that is confidently wrong. Nothing in that class may remain open when you pass.