"A human reviews it" often doesn't work
When review is a rubber stamp, you've got the risk without the safety.
Adding a review step is the standard answer to AI risk. Research explains why it often disappoints. A 2010 review of automation studies found that people over-rely on automated advice, that experts do it too, and that training or instructions don't prevent it. When output is right 95 times out of 100, people stop looking closely.
Make review real
- Show the source beside the output, so checking takes seconds.
- Point reviewers to what to check, such as the amount and the name.
- Limit the volume. Someone approving 300 items a day isn't reviewing them.
- Test the process now and then by adding a known error and seeing if it's caught.
- Give reviewers the authority and the time to say no.
If you can't make review work at the volume you need, change the design. Narrow what the system can do, or review only the risky cases carefully.
A question for your next review
Ask how many outputs reviewers changed or rejected last month. If the answer is close to zero, either the system is excellent or nobody's really looking. Seeding a few known errors will tell you which.
Sources: Parasuraman and Manzey (2010), Complacency and bias in human use of automation, Human Factors