A rule-based program is a set of instructions written by a person. The computer follows the instructions exactly, and the output is correct whenever the instructions are correct and complete.

Rule-based programming is the ordinary way of making a computer do something. For many tasks it is the right way, and Section 4 lists several. The difficulty appears with tasks that people can do easily but cannot describe as a list of instructions. Spam filtering is the standard example, and it is used throughout this lesson.

Worked example. A spam filter built entirely from written rules might develop as follows.

Each individual rule is easy to write. The problem is not difficulty but coverage: a rule-based filter requires its author to know every trick in advance, including tricks nobody has invented yet. The list of rules describes last month’s spam, and a sender who can test messages against the filter will always find the next gap.

Many problems have this shape. A person can tell junk mail from real mail in about a second, and cannot write down how the decision was made. The knowledge exists, but not in a form that can be turned into instructions.

The alternative is to write no rules at all and to supply examples instead. For the spam problem, that means gathering ten thousand emails that people have already sorted into two piles, spam and not spam, and handing the whole collection to a procedure.

A learning procedure is a program whose job is to find what the examples in one pile have in common that the examples in the other pile do not. It is not told what to look for. It searches, compares, and keeps whatever separates the piles best.

A model is what the learning procedure produces. Given a new example it has never seen, the model returns an answer. For the spam problem, the model is given a new email and returns a guess: spam or not spam.

Nobody wrote a rule about free money. If next year’s spam looks completely different, nothing is rewritten: newer examples are collected and the procedure is run again.

Figure 1 shows the most important feature of the second approach. The human work did not go away. It changed shape. Somebody still has to find ten thousand emails and sort them, and if that sorting is careless, everything built on it is careless too. Lesson 2 examines this sorting step in detail.

Machine learning is finding patterns in examples instead of following written instructions. The term means nothing more than that.

The word learning suggests more than actually happens, and the gap should be understood from the start.

A spam model that is right 98% of the time has not understood anything. It has not worked out that some people try to take money from strangers. What it holds is a set of statistical regularities: certain words appear far more often in one pile than the other, messages with many capital letters lean one way, links to unfamiliar addresses lean the same way. These regularities are useful. They are not comprehension.

The distinction predicts where such a system fails. A model is confident and wrong on anything unlike what it was shown. A model trained only on English email knows nothing whatsoever about Spanish spam. This is not a flaw in its reasoning; it was never shown any Spanish spam, and it has no way of reasoning its way there.

Every learning procedure in this course needs the same three things. Each is treated in its own lesson.

Remove any one of the three and no learning takes place.

Both approaches are correct tools, and choosing the wrong one wastes a great deal of effort. The test for choosing between them is short.

Write the rule when the rule can be stated exactly. Tax owed on an income is defined in law, so the calculation is written out and is exactly right every time. Whether a chess move is legal is written in the rules of chess. A thermostat that turns the heating on below 18 degrees needs no training data and no examples. A written rule is faster, costs nothing to run, can be checked by reading it, and never produces a surprise.

Learn from examples when a person can recognize the answer but cannot write down how. Nobody can state the rule that separates one person’s handwriting from another’s, or spam from a real message, or a photograph with a dog in it from one without. In these cases the only thing a person can do is point at cases and say this one, yes; that one, no.

Machine learning is not the more advanced choice. It is the choice for when the rule cannot be written down.

The first question to ask about any problem is therefore whether the rule can be written. If it can, write it. If all a person can do is point at examples, the problem belongs to machine learning, and the next step is to understand what those examples must look like before any procedure can use them. That is the subject of Lesson 2.