Compare · Optional exploration
Examine a confident classification
Twenty valid classifications report 90% confidence. Examine the evidence before using that score to route work.
Think it through with your agent
Give your agent this URL. It contains the scenario and guidance to ask one question at a time. Start with your own prediction or explanation, then check it against the book.
Use with your task. Compare one agent with the fleet. Keep discoveries in your existing evidence or learning notes.
Follow the argument. Check confidence before routing.
Read the exercise
Scenario#
A classifier must name the single layer affected by a proposed change, choosing interface or data.
In an invented policy-development batch of 20 single-layer changes, every output chooses interface with confidence 0.9 and passes schema validation.
The fixture's independently adjudicated reference labels are assumed correct: 12 are interface and eight are data.
These are invented teaching inputs, not observed model outputs.
Investigate#
- Calculate the observed reference agreement for this confidence group. Compare it with the reported confidence: what does each quantity measure, and what would you need before trusting the score on future changes?
- Suppose one change alters authorization and the project's policy requires human review. Does a valid label or a higher confidence score satisfy that requirement? Name the independent gate that should enforce it.
- Change the input: a diff crosses both layers, or the model receives an incomplete diff. What can neither label express, and when should the workflow abstain, retrieve more evidence, or escalate? Define that path even if the model returns a high score.
Use development cases to set the policy, then freeze it and evaluate on fresh held-out cases using the routing outcome and cost accounting. Record reference-label uncertainty, incorrect routes, abstentions, missing context, and changed task families; one confidence group does not describe calibration over every workload. Keep the findings in the existing evaluation notes.
Further reading#
Guo and colleagues' calibration paper defines the calibration question. The numbers here are invented teaching inputs, not measurements from that paper.
Show worked reasoning
All 20 outputs are valid choices, but only 12 match the reference labels: observed agreement is 12/20 = 60%, while reported confidence is 90%. The fixture exposes a gap between confidence and observed agreement in this group; it does not establish a future success probability or the model's overall calibration.
This is optional, ungraded exploration. The course brief defines the required work.