All starting points

Establish · Optional exploration

Compare evidence for an agent repair

Saving an event reports success, but the list stays unchanged. Compare task feedback, runtime logs, and a diagnostic brief.

Think it through with your agent

Give your agent this URL. It contains the scenario and guidance to ask one question at a time. Start with your own prediction or explanation, then check it against the book.

Read the agent starting point

Use with your task. Establish the evidence baseline. Keep discoveries in your existing evidence or learning notes.

Follow the argument. Use runtime evidence during repair.

Read the exercise

Scenario#

Invented paper fixture; no application or model has been run. An event tracker initially displays E1 and E2 in workspace W1. The actor may use W1; workspace W2 contains E9 and is forbidden. The contract requires a permitted saved event to survive restart and appear in the active workspace's unfiltered list before the interface reports completion, without a manual reload. A valid save returns 201; saving an event with an empty label returns 422 and must neither create nor display that event. A W2 request must return 403 without event data.

The browser reports that saving E7 succeeded, but the list still shows E1 and E2. An invented diagnostic packet contains these observations from one save and reload:

action A7: click Save for E7 in workspace W1
request R7: POST /workspaces/W1/events, event E7
response R7: 201, event E7
render after R7: workspace W1, event IDs E1 E2
manual reload: GET /workspaces/W1/events
reload response: 200, event IDs E1 E2 E7
render after reload: workspace W1, event IDs E1 E2 E7

No source line, storage-restart observation, or cause is supplied.

Investigate#

  1. What does the 201 response establish, and which observations help distinguish failure to store E7 from failure to display it? Name one explanation still consistent with the packet and the next check that would challenge it.
  2. Specify acceptance cases independently of a proposed fix: permitted saving and visibility before completion, persistence across restart, an empty-label save, and forbidden workspace access. State expected event IDs and responses from the contract, rather than deriving them from the implementation's own filter or success message.
  3. Plan three repair conditions, all receiving the same source, contract, initial failure report, tools, common checks, and repair budget. Give one no additional telemetry, one the raw diagnostic packet, and one a brief derived from that packet. Preserve the raw packet and record which observation supports each sentence in the brief. Keep additional diagnostic evidence within the assigned condition and record any deviation.
  4. For a real comparison, choose a small reproducible failure in a local application you control, using synthetic data and pinned revisions. The paper packet above cannot substitute for observations from that application. Verify that instrumentation preserves the failure, collect one shared packet, then start each repair in a fresh copy of the uninstrumented failing revision and a fresh agent context. Apply the same acceptance checks to every final patch without diagnostic instrumentation; preserve failures and unfinished attempts.
  5. Compare accepted behavior, repair time, model use, retries, and active human correction. Account separately for instrumentation, telemetry collection, brief preparation, and evidence review, including model calls used to summarize logs. With one trial per condition, report a case comparison; use repeated fresh trials and additional failures before claiming a reliable advantage.

You can complete the investigation on paper by describing checks and predicted outcomes. Label those as predictions, and mark unavailable execution or energy measurements as unmeasured. Use existing evaluation notes; the activity adds no required project feature or submission.

Further reading#

TeleGen, version 1 studies runtime evidence and repair briefs for generated web applications. Appendix A explains the comparison controls and limits.

Show worked reasoning

The 201 response records a successful HTTP outcome under the server's implementation. By itself it does not establish that the declared persistence contract was met or that the browser updated. The later GET returning E7 is evidence that the event is available through that read path at that time. The packet contains no storage restart, so durability across restart remains untested.

The render observations make the path from a successful response to the active list a useful place to investigate. They do not identify a source-level cause. A stale client cache, a missed invalidation, or a view using an earlier result could fit this record. Inspect the state transition and the data actually consumed by the list before selecting a repair. Retrying the POST merely to refresh the view could create another event while leaving the original cause unresolved.

A useful brief preserves the workspace, event and request identities, the successful response, the unchanged list, and E7's appearance after reload. It should label a suspected cause as a hypothesis. "The save failed" discards the later read; "the database is durable" invents an observation. Check each claim against the raw packet and retain gaps rather than filling them with a plausible story.

Acceptance belongs to the declared user and access behavior. For a valid E7, check that the list contains E1, E2, and E7 before reporting completion and after restart. An empty-label attempt must return 422 while leaving those stored and displayed IDs unchanged; a W2 request must return 403 without E9 or other event data. A fix that displays every event, shows rejected writes optimistically, or changes the test to require a reload has not preserved the contract. Instrumentation must also preserve the failure: logging can change timing or accidentally change code, making its evidence a different experiment.

Fresh copies and contexts prevent a later condition from inheriting an earlier fix or diagnosis. Equal repair budgets do not imply equal total cost: collecting telemetry and constructing the brief consume resources too. Record those costs even when one packet is reused, and distinguish shared setup from each trial's repair cost. One successful repair cannot establish that briefs reliably outperform raw logs or task feedback.

This is optional, ungraded exploration. The course brief defines the required work.