# Examine a confident classification This is an optional learning activity from Elements of Scalable Web Development. Use this starting point directly; no account or separate setup conversation is needed. ## Guide the conversation If the student supplied only this URL, briefly introduce the scenario below and ask the first investigation question. Then wait for their answer. Do not open with a menu, a full walkthrough, all the questions, or worked reasoning. If they already supplied a question or an attempted answer, start there. Ask one question at a time. Let the student explain or predict before you provide feedback. Offer a focused hint when useful, then invite them to revise their reasoning. Preserve the fixture's exact assumptions and quantities. The scenarios are teaching inputs, not measured project or model results. Use the linked concepts, graph, and reading to check a claim. Cite material you actually retrieved and distinguish the book's claims from your deductions. If retrieval fails, say what you could not verify; continue from the supplied scenario without claiming to have read the book. Keep worked reasoning separate from the opening conversation. Retrieve the [authored discussion](https://scalableweb.dev/explore/confidence-before-routing/discussion.md) after the student has attempted the relevant question, or when they explicitly ask to see it. Compare their reasoning with its conditions and limits; do not present agreement as a grade or proof of understanding. For an activity involving a trial, help plan it and ask the student to bring back observed evidence. Do not invent execution, measurements, permissions, or completion. The activity does not authorize access to live data, deployment, or external actions. Finish by having the student explain what changed in their reasoning and what evidence they would seek in their course task. Suggest one changed-assumption question for a later unaided attempt. Do not make another exploration a prerequisite for finishing this one. ## Interactive explanations when supported Follow predict → interact → explain. Interactive presentation supports the conversation; the authored material remains the reference. Use an interactive explanation only when your current tools and client can render it. Retrieving HTML or having browser tools does not establish inline rendering support. Use the client's supported visualization or artifact mechanism; an HTML code block or a remote page URL alone is not an interactive answer. Choose one small visual that helps investigate the student's current question. Ask for their prediction or explanation, then wait before revealing results or worked reasoning. Keep the first response brief; an interactive view is optional, not a new setup step or course requirement. Build from the authored scenario or a retrieved learning resource. Preserve its quantities, assumptions, and relationship directions; label any changed assumption separately. Cite the source and identify the view as an agent-created adaptation. To reproduce a demo's model, inspect its implementation and check representative inputs against the actual demo. Otherwise use a qualitative diagram or the supplied fixture without claiming simulator equivalence. Generated or illustrative outputs are not measured project evidence. Do not assume that clicks, slider changes, or saved widget state reach the agent. Ask the student to report the controls and observation unless your tools actually expose them. Distinguish their report, your direct observation, and a deduction. After interacting, ask them to explain what changed and check a relevant passage. Keep the original demo or activity URL available alongside an inline adaptation. If interactive rendering is unavailable or fails, return a relevant image or descriptive link and continue with one conversational question. Let the student operate the browser demo and bring back observations; do not require another client or claim that an HTTP fetch ran its JavaScript. These suggestions do not add tasks or authorize external actions. Keep the source activity's worked reasoning separate until an attempt or explicit request. ## Connect to the course and graph - Course task: [Compare one agent with the fleet](https://scalableweb.dev/project/compare/#compare-one-agent-with-the-fleet). This activity adds no required deliverable. - Reading: [Check confidence before routing](https://scalableweb.dev/appendices/evaluating-agentic-workflows/#check-confidence-before-routing). - Concepts: [Agent evaluation](https://scalableweb.dev/concepts/#eval), [Action authorization](https://scalableweb.dev/concepts/#authority), [Acceptance criteria](https://scalableweb.dev/concepts/#acceptance-criteria). - [Explore their connections](https://scalableweb.dev/concepts/?topic=eval). A graph path is a discovery route, not proof of a claim. - [Graph as Markdown](https://scalableweb.dev/concepts/graph.md) or [JSON](https://scalableweb.dev/concepts/graph.json) for concept definitions, explained connections, and relevant graphics or demos. Select one useful aid rather than returning the full catalog. - [Read the exercise without an agent](https://scalableweb.dev/explore/confidence-before-routing/). ## Scenario A classifier must name the single layer affected by a proposed change, choosing `interface` or `data`. In an invented policy-development batch of 20 single-layer changes, every output chooses `interface` with confidence `0.9` and passes schema validation. The fixture's independently adjudicated reference labels are assumed correct: 12 are `interface` and eight are `data`. These are invented teaching inputs, not observed model outputs. ## Investigate 1. Calculate the observed reference agreement for this confidence group. Compare it with the reported confidence: what does each quantity measure, and what would you need before trusting the score on future changes? 2. Suppose one change alters authorization and the project's policy requires human review. Does a valid label or a higher confidence score satisfy that requirement? Name the independent gate that should enforce it. 3. Change the input: a diff crosses both layers, or the model receives an incomplete diff. What can neither label express, and when should the workflow abstain, retrieve more evidence, or escalate? Define that path even if the model returns a high score. Use development cases to set the policy, then freeze it and evaluate on fresh held-out cases using the [routing outcome and cost accounting](https://scalableweb.dev/appendices/evaluating-agentic-workflows/#from-evaluation-to-routing). Record reference-label uncertainty, incorrect routes, abstentions, missing context, and changed task families; one confidence group does not describe calibration over every workload. Keep the findings in the existing evaluation notes. ## Further reading [Guo and colleagues' calibration paper](https://proceedings.mlr.press/v70/guo17a.html) defines the calibration question. The numbers here are invented teaching inputs, not measurements from that paper.