Lesson 3 of 5, 15 min with the exercise

Build a test set

Be able to tell whether a change made things better.

A test set is a list of real inputs with the right answers written down. 30 to 50 cases is plenty to start, as long as they're representative.

• Include the common cases in roughly their real proportions.
• Add the difficult ones you know about.
• Have a person who knows the work agree the right answers.
• Keep it separate from the examples you put in your prompt.

When a mistake turns up in live use, add it to the set. Over time, it becomes a record of everything that's gone wrong.

See it in practice

Spreadsheet format: Case ID, Input, Correct answer, Output, Pass or fail, Error type. For open-ended outputs such as summaries, write what a good answer must include and must not say, then have a person or a separate check score against that list.

Common mistake: Choosing only the cases where you know it works. A test set that always passes tells you nothing.