A decision tree is a model made of a sequence of yes-or-no questions about the features. Each answer leads either to another question or to a prediction. To make a prediction, start at the first question and follow the answers until a prediction is reached.

The nearest neighbor model from Grade 9 predicts by finding the stored example most similar to the new one and copying its label. Asked why it predicted a pass, the only answer it can give is that the new student resembles some other student. That is a true answer, but it does not tell a student what to change. A decision tree gives a different kind of answer: the list of questions and answers that led to the prediction.

Figure 1 shows a complete decision tree. It was trained on ten students, using the same two features that Grade 9 used: hours studied per week and attendance.

Worked example. A new student has 86% attendance and 7 hours of study per week. The tree is run as follows.

1. First question: is attendance 85% or more? 86% is, so the answer is yes. Go right.
2. Second question: is study time 3 hours or more? 7 hours is, so the answer is yes. Go right.
3. The right-hand branch ends in a prediction: pass.

The whole model was executed in two steps, and the reasoning can be stated in one sentence: attendance is at least 85% and study time is at least 3 hours, so the tree predicts a pass.

A model is interpretable when a person can inspect it and understand why it produces the answers it does. Decision trees are among the few models that are genuinely interpretable, not merely small.

Nobody chose the two questions in Figure 1 by hand. The training algorithm found them, and the method is simple: try every feature at every sensible threshold, and keep whichever split separates the labels most cleanly.

A split is one yes-or-no question about one feature at one threshold, such as “attendance ≥ 85”. A split divides the examples that reach it into a yes-side and a no-side.

The ten students the tree was trained on are listed below.

Four students passed and six failed. A candidate split is scored by one measure: the number of students that would be classified correctly if the tree stopped at that split and predicted the majority label on each side.

Worked example. Two candidate first questions, scored on the ten students.

For the attendance split, the yes-side majority is pass (4 of 5 correct) and the no-side majority is fail (5 of 5 correct), for 9 of 10 in total. For the hours split, the yes-side majority is pass (3 of 4 correct) and the no-side majority is fail (5 of 6 correct), for 8 of 10. The attendance split scores higher and is chosen.

A group is pure when every example in it has the same label. A pure group needs no further questions, because predicting its one label gets every example right.

The reason the attendance split is the better choice matters as much as its score. Its no-side is completely pure: five students, all of whom failed, with no exceptions. That is the real target of the algorithm. A split is good when it produces groups that lean as far as possible toward one label.

The algorithm then repeats the same procedure on any group that is still mixed.

1. The no-side of the attendance split is pure, so it becomes a prediction: fail.
2. The yes-side holds four passes and one failure, so a second question is chosen for those five students only.
3. The split hours ≥ 3 separates them perfectly: the four students with 4, 8, 10 and 12 hours passed, and the one student with 2 hours failed.
4. Every group is now pure, so the tree is finished. It classifies all ten training students correctly.

The tree in Figure 1 classifies all ten training students correctly. Grade 9 Lesson 5 established what a perfect training score usually means: the model may have memorized the training data rather than learned a pattern. The tree version of this problem is the most direct one in machine learning.

A tree can always reach 100% on its training data. If any group is still mixed, the algorithm asks another question. If that is allowed to continue, eventually every group holds exactly one student, and every student is classified correctly by the question that isolated them. The result is a lookup table written as a set of questions.

The left panel of Figure 2 is the tree in Figure 1: two cuts, which can be described in a sentence. The right panel is the same algorithm allowed to keep splitting. Each extra box exists to accommodate one student. Those boxes are not patterns; they are individual students, memorized. A new student who lands in one of those boxes gets the label of whichever training student happened to be there.

Maximum depth is the number of questions a tree may ask before it must commit to an answer. Depth 1 is a single question. Depth 2 is the tree in Figure 1. Depth 10 is the right-hand panel of Figure 2.

Maximum depth is the tree’s equivalent of k in nearest neighbor: a setting that controls how closely the model fits its training data. It is chosen the way Grade 9 chose k. Try several values, and keep the one that scores best on data the tree never saw. The choice is never made on the training rows, because training accuracy only ever rises as depth increases.

Any setting that makes training accuracy go up is a setting that can tune a model into uselessness.

Decision trees have three practical strengths.

1. They explain themselves. For any decision where a person is entitled to know the reason, such as a loan, a diagnosis or a place on a course, a model that can be printed out and read is worth a great deal.

2. They do not depend on units or scale. Grade 9 Lesson 3 required every feature to be rescaled to 0–1, because attendance measured on 0–100 drowned out hours measured on 0–20 inside the distance calculation. A tree never compares two different features with each other. It asks whether attendance is above 85, which is a question about attendance alone. Rescaling a feature changes the threshold number but not the students on each side, so rescaling changes nothing about the tree’s predictions.

3. They take categories directly. “Is the subject history?” is already a yes-or-no question. No conversion of the category to a number is needed.

The limitation is visible in Figure 2. Every cut a tree makes is straight across one axis, because every question is about one feature. A tree’s boundaries are therefore always horizontal and vertical lines. The overall shape is a staircase.

A staircase is adequate until the real pattern runs diagonally. Suppose what matters is hours and attendance together, so that the true boundary is a slanted line. A tree can approximate the slant, but only with many small steps, and each step costs a question and uses up training data. The tree is doing with twenty questions what a single tilted line would do with one.

The next model draws the tilted line directly. Lesson 2 builds one, and obtains something a tree does not provide: not just an answer, but a measure of how confident the model is in it.