Lesson 5 of 5, 11 min with the exercise

Improve it, or stop

Make the decision on evidence.

Change one thing at a time, then rerun the full test set. If you change the prompt and the model together, you won't know which one helped. Rerunning everything also catches fixes that break something else.

Decide in advance what would make you stop. For example: "If the cost per task isn't lower after 30 days, or if any serious error reaches a customer, we switch it off and review."

Stopping a project that didn't work is a useful result. You've spent a month finding out, and you have numbers to show for it.

See it in practice

Sample decision rule: "Expand if cost per task is at least 30% lower for 30 days and no high-severity error reaches a customer. Fix and retest if savings are there but errors aren't acceptable. Stop if neither improves after two rounds." Your thresholds will differ. What matters is writing them down first.

Common mistake: Moving the goalposts after seeing the results. Once time has been spent, there's pressure to call any result a success.