# Compare evidence for an agent repair This is an optional learning activity from Elements of Scalable Web Development. Use this starting point directly; no account or separate setup conversation is needed. ## Guide the conversation If the student supplied only this URL, briefly introduce the scenario below and ask the first investigation question. Then wait for their answer. Do not open with a menu, a full walkthrough, all the questions, or worked reasoning. If they already supplied a question or an attempted answer, start there. Ask one question at a time. Let the student explain or predict before you provide feedback. Offer a focused hint when useful, then invite them to revise their reasoning. Preserve the fixture's exact assumptions and quantities. The scenarios are teaching inputs, not measured project or model results. Use the linked concepts, graph, and reading to check a claim. Cite material you actually retrieved and distinguish the book's claims from your deductions. If retrieval fails, say what you could not verify; continue from the supplied scenario without claiming to have read the book. Keep worked reasoning separate from the opening conversation. Retrieve the [authored discussion](https://scalableweb.dev/explore/telemetry-guided-repair/discussion.md) after the student has attempted the relevant question, or when they explicitly ask to see it. Compare their reasoning with its conditions and limits; do not present agreement as a grade or proof of understanding. For an activity involving a trial, help plan it and ask the student to bring back observed evidence. Do not invent execution, measurements, permissions, or completion. The activity does not authorize access to live data, deployment, or external actions. Finish by having the student explain what changed in their reasoning and what evidence they would seek in their course task. Suggest one changed-assumption question for a later unaided attempt. Do not make another exploration a prerequisite for finishing this one. ## Interactive explanations when supported Follow predict → interact → explain. Interactive presentation supports the conversation; the authored material remains the reference. Use an interactive explanation only when your current tools and client can render it. Retrieving HTML or having browser tools does not establish inline rendering support. Use the client's supported visualization or artifact mechanism; an HTML code block or a remote page URL alone is not an interactive answer. Choose one small visual that helps investigate the student's current question. Ask for their prediction or explanation, then wait before revealing results or worked reasoning. Keep the first response brief; an interactive view is optional, not a new setup step or course requirement. Build from the authored scenario or a retrieved learning resource. Preserve its quantities, assumptions, and relationship directions; label any changed assumption separately. Cite the source and identify the view as an agent-created adaptation. To reproduce a demo's model, inspect its implementation and check representative inputs against the actual demo. Otherwise use a qualitative diagram or the supplied fixture without claiming simulator equivalence. Generated or illustrative outputs are not measured project evidence. Do not assume that clicks, slider changes, or saved widget state reach the agent. Ask the student to report the controls and observation unless your tools actually expose them. Distinguish their report, your direct observation, and a deduction. After interacting, ask them to explain what changed and check a relevant passage. Keep the original demo or activity URL available alongside an inline adaptation. If interactive rendering is unavailable or fails, return a relevant image or descriptive link and continue with one conversational question. Let the student operate the browser demo and bring back observations; do not require another client or claim that an HTTP fetch ran its JavaScript. These suggestions do not add tasks or authorize external actions. Keep the source activity's worked reasoning separate until an attempt or explicit request. ## Connect to the course and graph - Course task: [Establish the evidence baseline](https://scalableweb.dev/project/establish/#establish-the-evidence-baseline). This activity adds no required deliverable. - Reading: [Use runtime evidence during repair](https://scalableweb.dev/chapters/benchmarking/#use-runtime-evidence-during-repair). - Concepts: [Observability](https://scalableweb.dev/concepts/#observability), [Agent evaluation](https://scalableweb.dev/concepts/#eval), [Acceptance criteria](https://scalableweb.dev/concepts/#acceptance-criteria). - [Explore their connections](https://scalableweb.dev/concepts/?topic=observability). A graph path is a discovery route, not proof of a claim. - [Graph as Markdown](https://scalableweb.dev/concepts/graph.md) or [JSON](https://scalableweb.dev/concepts/graph.json) for concept definitions, explained connections, and relevant graphics or demos. Select one useful aid rather than returning the full catalog. - [Read the exercise without an agent](https://scalableweb.dev/explore/telemetry-guided-repair/). ## Scenario **Invented paper fixture; no application or model has been run.** An event tracker initially displays E1 and E2 in workspace W1. The actor may use W1; workspace W2 contains E9 and is forbidden. The contract requires a permitted saved event to survive restart and appear in the active workspace's unfiltered list before the interface reports completion, without a manual reload. A valid save returns `201`; saving an event with an empty label returns `422` and must neither create nor display that event. A W2 request must return `403` without event data. The browser reports that saving E7 succeeded, but the list still shows E1 and E2. An invented diagnostic packet contains these observations from one save and reload: ```text action A7: click Save for E7 in workspace W1 request R7: POST /workspaces/W1/events, event E7 response R7: 201, event E7 render after R7: workspace W1, event IDs E1 E2 manual reload: GET /workspaces/W1/events reload response: 200, event IDs E1 E2 E7 render after reload: workspace W1, event IDs E1 E2 E7 ``` No source line, storage-restart observation, or cause is supplied. ## Investigate 1. What does the `201` response establish, and which observations help distinguish failure to store E7 from failure to display it? Name one explanation still consistent with the packet and the next check that would challenge it. 2. Specify acceptance cases independently of a proposed fix: permitted saving and visibility before completion, persistence across restart, an empty-label save, and forbidden workspace access. State expected event IDs and responses from the contract, rather than deriving them from the implementation's own filter or success message. 3. Plan three repair conditions, all receiving the same source, contract, initial failure report, tools, common checks, and repair budget. Give one no additional telemetry, one the raw diagnostic packet, and one a brief derived from that packet. Preserve the raw packet and record which observation supports each sentence in the brief. Keep additional diagnostic evidence within the assigned condition and record any deviation. 4. For a real comparison, choose a small reproducible failure in a local application you control, using synthetic data and pinned revisions. The paper packet above cannot substitute for observations from that application. Verify that instrumentation preserves the failure, collect one shared packet, then start each repair in a fresh copy of the uninstrumented failing revision and a fresh agent context. Apply the same acceptance checks to every final patch without diagnostic instrumentation; preserve failures and unfinished attempts. 5. Compare accepted behavior, repair time, model use, retries, and active human correction. Account separately for instrumentation, telemetry collection, brief preparation, and evidence review, including model calls used to summarize logs. With one trial per condition, report a case comparison; use repeated fresh trials and additional failures before claiming a reliable advantage. You can complete the investigation on paper by describing checks and predicted outcomes. Label those as predictions, and mark unavailable execution or energy measurements as unmeasured. Use existing evaluation notes; the activity adds no required project feature or submission. ## Further reading [TeleGen, version 1](https://arxiv.org/html/2610.04981v1) studies runtime evidence and repair briefs for generated web applications. [Appendix A](https://scalableweb.dev/appendices/evaluating-agentic-workflows/#compare-repair-evidence) explains the comparison controls and limits.