Build an AI quality dashboard

What you'll build

A simple dashboard, in a spreadsheet or your BI tool, tracking volume, quality, cost and failures for a live AI workflow.

The steps

1. 1
   Log every run

   Store the time, input ID, output, model version, cost, and whether it passed checks.

2. 2
   Record human edits

   Save whether the reviewer accepted, changed or rejected each output.

3. 3
   Sample for quality

   Each week, have someone score a random 20 items.

4. 4
   Chart four numbers

   Volume, acceptance rate, cost per task and failed runs.

5. 5
   Set alerts

   Alert on failed runs, on no runs at all, and on a sudden drop in acceptance.

6. 6
   Review it monthly

   Agree what you'll change, and note any model or prompt changes on the chart.

Practice material

Log format, one row per run:
run_id, time, workflow, input_id, model_version, prompt_version,
cost, duration, status, passed_checks, reviewer_action

Sample rows (invented):
r-1001, 2026-03-02 09:01, triage, t-552, m1, p3, 0.01, 2.1s, ok, yes, accepted
r-1002, 2026-03-02 09:04, triage, t-553, m1, p3, 0.01, 2.4s, ok, no, changed
r-1003, 2026-03-02 09:06, triage, t-554, m1, p3, 0.00, 30s, timeout, no, none

You're done when

• You can answer "is it still working?" in a minute
• A deliberate failure sets off an alert
• Each change in the numbers can be matched to a date and a version