Lesson 4 of 5, 11 min with the exercise
Design the human review
Make review quick enough that people really do it.
If reviewers have to reopen three systems, they'll start clicking approve without looking. Put the AI's output, the source and an approve or edit button on one screen.
- Review everything at first.
- As results hold up, move to reviewing only low-confidence and high-value items, plus a random sample.
- Record every edit. Those edits show you what to fix next.
Keep the random sample even when things look good. It's how you notice quality slipping when the inputs change.
See it in practice
A useful test: time your reviewers on 20 items, and add one or two deliberate errors. If they're much faster than checking should allow, or they miss the errors, your design is relying on a check that isn't really happening.
Common mistake: Measuring reviewers only on speed. If the target is items per hour, approval becomes the job and checking gets squeezed out.
Try it
Sketch what a reviewer would see for one item. If it takes more than a glance to find the source, redesign it.
You're done when
- Output and source appear on one screen
- Edits are recorded
- A random sample stays in place
Sources: Parasuraman and Manzey (2010), Complacency and bias in human use of automation, Human Factors