Elements of Scalable Web Development

Appendix A

Evaluating agentic development workflows#

Created
Updated

Use this appendix to design, run, or critique an agentic-development evaluation. The Benchmarking chapter defines the course baseline.

A.1 Apply the method selectively#

Begin with representative tasks, checked outcomes, preserved failures, and a declared workflow configuration. Read evaluation anatomy, task selection, and grader design for that minimal path. Add other sections when the decision requires them:

A.2 Anatomy of an agent evaluation#

A software test usually invokes known code with controlled inputs and checks the result. An agent evaluation must also define the situation in which a model acts, because tool access, instructions, the starting repository, resource limits, and intermediate state can all change the result. The following vocabulary helps keep those layers separate (Grace et al., 2026): The table below locates each term within one evaluation.

Table A.1. Vocabulary for the nested objects and responsibilities in an agent evaluation. #
Term Unit or scope What it records or does Common confusion
Task One declared problem Defines inputs and success criteria A repeated attempt is not a new independent task
Trial One attempt at one task Runs the task once so repeated trials can expose behavioral variation Trial count is not task diversity
Outcome End state of one trial Preserves the resulting patch, database state, generated artifact, or other effect The final claim is not necessarily the actual outcome
Trace or transcript Path through one trial Records outputs, tool calls, intermediate results, and relevant interventions A plausible trace does not prove a correct outcome
Grader One evaluated property Checks an aspect of the outcome or trace through code, a model-based rubric, or human judgment One grader is not the whole evaluation
Agent harness Model-and-tool execution loop Supplies context, tools, state, and control for one agent run It need not provision or score the evaluation
Evaluation harness Evaluation infrastructure Provisions tasks and environments, invokes the agent harness, preserves evidence, applies graders, and aggregates results It is distinct from the agent being evaluated
Evaluation suite A declared set of tasks Groups tasks intended to measure a capability or behavior A shared benchmark is one kind of suite

These terms describe different measurement objects. A repository test is often one deterministic grader inside an agent evaluation rather than the entire evaluation. A public benchmark is a shared evaluation suite designed for comparison across systems, while a local product or project evaluation should represent the work and failure modes that matter in its own setting. Treating these as synonyms hides differences in task selection, measurement, and supported claims.

Prefer grading the outcome when several valid paths can solve the task. Use the trace to diagnose failures, account for cost and intervention, and check process requirements that genuinely matter, such as permission compliance or a mandatory approval. Rigidly requiring one anticipated sequence of tool calls can reject a valid solution, while accepting only the final statement can miss that the claimed change never occurred.

A.3 Capability and regression evals#

A capability eval asks whether the workflow can perform work that is currently difficult. Its initial pass rate may be low because it is intended to reveal a boundary and provide room for improvement. A regression eval asks whether the workflow still handles previously accepted work and should normally pass almost all trials (Grace et al., 2026). When a capability task becomes reliably solvable, preserve it as a regression task rather than discarding the evidence and starting over.

Start with tasks already checked manually, consequential failures, and representative project changes. For each task, keep a reference solution or known passing run to show that the stated problem is solvable and the graders can accept at least one correct outcome. Everything required by a grader should be justified by the task brief or a declared invariant; an agent should not fail because the evaluator silently assumed a filepath, output format, or environmental resource. Balance positive and negative cases so optimizing one behavior does not produce its opposite failure elsewhere.

A.3.1 Probe an API with temporary clients#

Lemire (2026) proposes building disposable applications on top of a maintained component to probe its behavior and API usability. This is integration testing through temporary consumers; the essay reports practitioner experience rather than measured defect-detection gains. Keep the provider and its public contract fixed while building the clients, and investigate whether a failure belongs to the provider, consumer, environment, documentation, or acceptance check. Preserve confirmed reproductions and regression cases before discarding the client scaffolding.

The optional API-consumer investigation uses a synthetic paginated event feed, a CSV client, and a summary client. Predict what their results establish, then design independent checks and reduce any failure before deciding which component to change.

A.3.2 Evaluate the cost of the next change#

An initial pass does not show how easily an implementation can be extended. SlopCodeBench addresses this by asking agents to extend their own solutions through successive specification checkpoints, leaving internal structure open and measuring structural erosion and redundant code alongside functional correctness (Orlanski et al., 2026). Preserving the implementation lets earlier decisions affect later work. Its structural measures are diagnostic proxies; human comprehension and maintenance effort still need their own evidence.

Consider two implementations, X and Y, of a small parcel-quotation tool. Each receives the same three changes in the same order:

  1. Quote one domestic parcel using a fixed rate and reject invalid weights.
  2. Accept a batch, preserving input order and reporting invalid parcels individually.
  3. Accept configurable rates while preserving the original rate when configuration is omitted.

The following numbers are invented teaching data, not SlopCodeBench results or measurements from a course project. Suppose the fictional trace shows X duplicating rate logic in its batch path, while Y reuses the single-parcel calculation. At checkpoint 3, X accidentally removes the default from the batch path, breaking a previously passing check. Both implementations eventually pass all current and retained acceptance checks after any repairs.

The table below records what that final pass alone would hide. Each paired cell gives X / Y; times are active human minutes for that checkpoint, with review and repair recorded separately. Agent spend includes every attempt and retry through acceptance, including calls made during repairs, but excludes human time.

Table A.2. Invented checkpoint evidence for two implementations that eventually pass the same acceptance checks. #
Checkpoint Regressions found before repair Review minutes Repair minutes Agent spend (USD)
1 — Single quote 0 / 0 6 / 6 0 / 0 0.40 / 0.40
2 — Batch quotes 0 / 0 12 / 8 0 / 0 0.80 / 0.70
3 — Configurable rates 1 / 0 24 / 9 18 / 0 2.40 / 0.90

Both implementations finish three checkpoints, but X consumes 60 human minutes of review and repair and USD 3.60 in agent spend; Y consumes 23 minutes and USD 2.00. The fictional trace suggests where to investigate the difference: a duplicated rule made a later change harder to propagate. A real comparison would still need to distinguish architecture from task difficulty, reviewer familiarity, model variation, and other changed conditions. Fewer lines or duplicated blocks would not establish the explanation by themselves.

Use the task and grader controls with a frozen change sequence, served configuration, per-checkpoint budgets, and stopping rules. Retain revisions, checks, pre-repair failures, acceptance decisions, human effort, and all-attempt cost, including unfinished trajectories. Checkpoints depend on earlier work; repeat fresh trajectories to study variation rather than counting checkpoints as independent trials.

This optional design studies successive changes within one implementation. The course's A–B–C builds start fresh, while cross-team probes examine another group's ability to understand and change each build.

For an optional repair comparison, branch X after checkpoint 2 and consolidate its duplicated rate rule in one variant while retaining the other. Check that both variants still meet the same baseline behavior, then give each the same later changes, including one that exercises the repaired boundary. Count cleanup and compatibility work alongside subsequent review, repair, regressions, and agent spend; a faster later change may not repay the initial intervention. Also record whether the duplication returns during a later change. A third variant can enforce the new dependency boundary to test whether that extra constraint prevents recurrence. The architectural-degradation review maps possible diagnostics and interventions, but its catalogue does not establish which repair will improve a particular system (Ahmad et al., 2026).

A.4 Evaluate assistance and retained competence#

Use this optional comparison when learning or retaining a skill is part of the task's purpose. The Scalable Development chapter distinguishes accepted output, retained competence, and valued participation. Evaluate these outcomes separately: a behavioral check cannot establish what the person can do unaided or which activities they want to retain.

The preprint by Shen & Tamkin (2026) reports a randomized study of 52 programmers learning the unfamiliar Trio library. AI access reduced immediate understanding scores without a statistically significant average completion-time benefit. Its observed interaction patterns were not randomly assigned, so the study does not establish that asking for explanations causally protects learning. A short library-learning session also cannot establish long-term professional skill loss.

In a classroom trial with nearly a thousand high-school mathematics students, Bastani et al. (2025) found that unrestricted AI access improved assisted practice but reduced subsequent unaided performance. A tutor using teacher-designed safeguards largely avoided that harm without establishing a positive unaided effect. The mathematics tasks and tutoring design limit transfer to university programming.

The preprint by Contractor & Reyes (2026) supplies a positive comparison: undergraduates randomly assigned AI access while studying unfamiliar topics performed better on an immediate unaided test. The estimated advantage one week later was smaller and less statistically certain. Students chose how to use AI; the reported differences between tutoring and outsourcing therefore do not establish a causal effect of either mode. This short topic-learning experiment does not measure long-term software-engineering expertise. Across these different tasks and assistance designs, the useful question is which support helps a particular learner retain and apply knowledge; neither inevitable skill loss nor automatic learning gains follow from AI access alone.

In a personal essay, Stewart (2026) describes missing the investigation and decisions involved in implementation even when the result passes his checks. Mollick (2026) offers a complementary perspective: human knowledge, taste, and initiative can turn agent capability into projects a person would otherwise struggle to attempt. These accounts motivate choices about delegation; they do not establish a universal effect on skill or well-being.

A.4.1 Optional exercise — Write with an agent as navigator#

The Navigator Agent pattern assigns implementation to the human while the agent provides guidance. It is a practitioner proposal, not evidence that typing suggested code preserves understanding. Use it to explore which assistance helps you retain a skill you need to own.

  1. Choose two small, comparable tasks involving an unfamiliar part of the stack, such as parsing an API response or enforcing a lifecycle transition. State the required behavior and give each task the same time budget.
  2. For one task, write the code yourself while the agent offers hints, explanations, and critique. Predict the next step before requesting a hint, and record where guidance changed your reasoning.
  3. For the other task, let the agent implement the change and use the remaining time to review and correct it. Apply the same kinds of acceptance checks to both results.
  4. In a later session, without the agent or transcript, explain each implementation's invariant and repair a small, previously unseen fault. Record correctness, explanation quality, time, and assistance needed during the original work.

Compare accepted output with later unaided performance; code volume and keystrokes measure neither understanding nor retained judgment. Task differences and practice order can explain a result from one pair, so use this as a learning diagnostic rather than a causal study.

As an optional reflection in your learning diary or discussion, name an activity you wanted to retain, whether the assistance mode preserved it, and what mode you would choose next time. Your preference is separate from the behavioral checks and later unaided performance.

A.5 Evaluate a planning interface#

Appleton (2026) proposes shared plans with inspectable alternatives, working prototypes, and direct editing to help people reason about agent work. Her examples and the related Chopin prototype illustrate a design direction; they do not establish that richer interfaces improve decisions, and the visual explanations required substantial author guidance.

For an optional comparison, choose one bounded decision, such as where to enforce retry deduplication. Prepare a text explanation and an inspectable state diagram or small prototype from the same starting revision, alternatives, assumptions, and checks. Hold the available evidence and preparation budget explicit; label any behavior that is only illustrated rather than executed. An interactive view should let the reviewer examine the consequence of a changed assumption, such as a crash between recording an effect and acknowledging it.

Before revealing a preferred option, ask the reviewer to explain each alternative's failure case, select an approach, and state what evidence would change that choice. Then introduce a changed requirement and ask them to revise the decision. Record misunderstandings, justified revisions, unresolved assumptions, review time, and the cost of producing each presentation. Keep decision quality separate from visual appeal and approval speed. Comparisons across people or tasks need balanced assignment and independent scoring; showing both views of the same case to one person introduces learning and order effects.

A.6 The eval suite is also software#

Terminal-Bench provides a concrete example of a versioned agent benchmark whose tasks combine instructions, container environments, resource limits, reference solutions, and executable tests. Terminal-Bench 2.1 changed 26 tasks from the earlier suite to repair bugs, adjust timeouts or resources, and improve robustness against reward hacking (Terminal-Bench Contributors, 2026). Its fix-code-vulnerability task, for example, pins a Bottle source revision and Python image, specifies CPU, memory, storage, and time limits, requires both a patch and a structured report, and checks the result with original and security-focused tests.

The measurement instrument can be wrong. Ambiguous instructions, incomplete graders, leaked state, shared caches, resource contention, and permissive reference matching can create failures or passes unrelated to the intended capability. Record the suite version and environment with every result, validate a reference solution, start trials from clean state, and read selected traces before trusting an aggregate score. When a task or grader changes materially, do not silently compare the new score with results from the previous instrument.

For this course, inspecting one packaged benchmark task is more useful than running the entire public suite. Identify its claimed capability, starting state, visible instructions, hidden or external assumptions, graders, reference solution, resource boundary, and plausible gaming paths. Then state what a passing result would and would not establish.

Some outcomes can be checked deterministically through tests, schemas, or expected artifacts. Others require a rubric or human judgment. When a model acts as a judge, test whether answer order, verbosity, confidence, or the judge's own style changes its scores; LLM judges have documented position and self-enhancement biases (Zheng et al., 2023). Keep a small human-reviewed calibration set and report disagreement rather than treating the judge as ground truth.

If a judge guides development, reserve independent acceptance evidence for final evaluation. A held-out task set does not make the scoring process independent when the same judge design also supplies the reported result.

A.6.1 Calibrate a rubric on actual attempts#

Freeze one task and rubric before looking at the outputs. For a short explanation task, the following authored input can be given unchanged to two permitted model attempts:

Context: Workspace(id), WorkItem(id, workspace_id, state),
ActivityEvent(id, workspace_id, work_item_id, actor, occurred_at).
Items move open -> in_progress -> done. Each accepted creation or
transition must record one matching event; rejection changes neither
state nor successful-event history. No implementation code is supplied.

Explain the entities and relationships, distinguish accepted and rejected
actions, and identify one enforcement question this context cannot settle.
Cite the supplied context and do not invent an implementation guarantee.

Use Table A.3 for both actual outputs. Score 0 for absent or incorrect, 1 for partial, and 2 when the stated criterion is met; quote an excerpt or explain the missing evidence for each judgment.

Table A.3. A fixed rubric for a bounded explanation task, independent of the model identity. #
Criterion Evidence for a score of 2
Grounding Distinguishes the three entities and their relationships using only the supplied context.
Behavior Explains the allowed lifecycle and that accepted actions require one matching event while rejected actions change neither record.
Uncertainty Identifies an unresolved enforcement question, such as atomic persistence, without asserting that the implementation guarantees it.

Label the actual responses X and Y; when another person prepares them, conceal configuration identities until scoring is complete. Keep the exact prompt, response, model and harness configuration, source record, corrections and acceptance decision. Record elapsed time and cost only when their observation boundary and source are available; missing observations remain unknown. Compare criterion-level reasons before adding scores, and preserve a disagreement instead of treating an authored answer as the only acceptable wording. Two attempts establish a comparison of those outputs, not a ranking of models or a reliable estimate of future success.

For an executable task, use one exercise in the course workspace, such as csv-export/, and keep its fixed acceptance checks separate from the manual rubric. Use a disposable clean copy for each attempt, retain failures and human corrections, and check actual output behavior rather than only the explanation. This is optional practice within existing study time. Without model access, analyze the instructor-supplied records used by the Compare activity, retaining their stated provenance and missing evidence; do not label an invented answer as an actual run.

Modern Web Guidance provides a concrete example of evaluating an imported agent capability. Its evaluation harness first checks browser assertions against both a reference implementation and a deliberately flawed implementation, then compares agent runs with and without the guidance. A project can reproduce the smaller experimental shape: hold the task, model, starting repository, and acceptance checks constant; change whether the capability is available; and compare correctness, retries, cost, and human correction. The resulting difference measures performance on that task and those checks, not whether the guidance makes every application better in production.

Literal task completion is not enough for an agent that can take actions. Score task success, permission compliance, unnecessary actions, side effects, and clarification quality separately.

For the course project, evaluate one representative AI-assisted task twice: once for the correctness of the produced artifact and once for the quality of the workflow that produced it. Preserve the prompt or task brief, relevant context, model and tool versions, execution trace, checks, human corrections, and final acceptance decision.

A.7 Research behind loop engineering#

Loop engineering connects several research questions: how agents use feedback, what progress survives another attempt, how runs are initiated and bounded, and what the resulting work costs people. The practitioner sources introduce an emerging label; scientific studies examine different parts of the design rather than establishing one universally effective recipe.

The table below distinguishes direct research on the named practice from earlier work on its mechanisms.

Table A.4. Scientific sources on loop engineering and related feedback, memory, and repair mechanisms. #
Research What it examines What the evidence supports and leaves open
Loop Engineering: Building Blocks, Adoption, and Impact (Lulla et al., 2026) An exploratory literature review and repository-mining study; version 2 is an under-review preprint. The authors confirmed autonomous loops in 217 of 256 heuristic-matched repositories; detection can miss other designs and the controlled experiment on effort and outcomes remains proposed.
Self-Refine (Madaan et al., 2023) The same model generates an output, critiques it, and revises it; evaluated on seven tasks at NeurIPS 2023. Reports improvements over one-step generation in the tested settings, without establishing the value of recurring unattended repository work.
Reflexion (Shinn et al., 2023) Feedback and textual reflection retained across trials, evaluated on decision-making, reasoning, and coding tasks at NeurIPS 2023. Supports studying memory and feedback as design variables, without establishing production maintainability or developer-effort savings.
Is Self-Repair a Silver Bullet for Code Generation? (Olausson et al., 2024) Repair experiments with Code Llama, GPT-3.5, and GPT-4 on HumanEval and APPS; accepted at ICLR 2024. Cost-aware gains were variable and sometimes absent, while stronger feedback improved repair; iteration count alone cannot justify a repair policy.

These studies concern different units: a revised answer, repeated attempts at a task, or recurring repository runs. Treat them as related evidence and keep their task, model, budget, and outcome boundaries visible when transferring a claim. In particular, adoption counts cannot establish productivity, and a benchmark improvement cannot establish lower human review or maintenance cost.

For a local comparison, hold the task, starting revision, model and tools, acceptance checks, and resource limits stable while changing one loop decision. Record accepted outcomes, unsuccessful attempts, elapsed time, resource use, and human correction or review. The uncertainty guidance explains why one successful task remains a case comparison.

A.7.1 Check retained workflow state#

In a practitioner account, Niska (2026) maintains current decisions, open questions, plans, and blockers outside the coordinator's conversation. Completed work leaves that record when Git or project documentation owns it. The account motivates testing continuity across sessions, but does not establish that another coordination layer saves effort.

For a bounded trial, resume the same synthetic task from a short current-state record and from the existing handoff method. Include an obsolete assumption and a blocker that has since cleared; check whether the resumed agent consults the authoritative source before acting. Measure repeated work, missed constraints, correction effort, and all context/retrieval cost. Keep standing authority in the project's owning instructions: a remembered plan is evidence of an earlier decision, not permission to widen the task.

A.8 Design the verification loop#

The Scalable Development chapter introduces loop engineering; this section develops its verification decisions. Begin with a claim about behavior and describe a plausible wrong outcome before choosing the check. For example, suppose a CSV export must contain only events the requesting user may read. A downloaded file can have valid syntax while including another workspace's events; checking that the download occurred would miss that fault. Define the expected event identities and permitted fields from the task contract before implementing the exporter.

The table below compares checks for that example and an exporter replacement. Use the combination that resolves the uncertainty in the change.

Table A.5. Complementary checks for a CSV exporter and the claims their results support. #
Check Example observation Evidence limit
End-to-end Seed synthetic permitted and forbidden events, export through the interface, and inspect the downloaded rows against independently specified expectations. Establishes the asserted behavior for the exercised users, data, and environment.
Property-based Generate event values and permission assignments; check that exported identities stay within the permitted set and field values survive CSV quoting. Depends on the correctness of the properties and the cases the generators explore.
Differential For a replacement exporter, compare parsed old and new output on the same inputs, allowing only documented intentional differences. Establishes compatibility on those inputs; the old exporter may contain defects.

For differential checks, normalize only differences the contract makes irrelevant; compare raw output too when byte-level compatibility matters. Investigate mismatches before deciding whether to repair a regression or document an intended behavior change.

A.8.1 Challenge the check itself#

Executing a check separately does not make its assumptions independent of the implementation. An expected result calculated with the exporter's own access filter can reproduce its bug. Use separately specified fixtures or a simpler reference rule, and inspect generated tests alongside generated code. For a consequential check, state one named fault it should detect and confirm that it fails when that fault is introduced reversibly. The concern-directed mutation lab also checks that a behavior-preserving implementation change still passes. This challenges both false acceptance and tests coupled too tightly to one implementation. A fresh reviewer can help identify assumptions, but agreement between reviewers still needs behavioral evidence.

The TestJack preprint supplies a research example (Yao et al., 2026). Its detailed audit reports refuting 1,545 of 4,487 previously passing trials across five coding benchmarks, or 34.4% of those passing trials. The auditors derive checks from task requirements, require a replayable test that passes the reference and fails the candidate, and use a separate reviewer to challenge the requirement. That percentage describes the selected audit corpus, not all agent-produced software.

An executable difference still needs adjudication: the reference can be wrong, or the check can assert an incidental choice rather than a requirement. Preserve disputed expectations alongside the witness. If an audit changes the acceptance suite, version it and re-evaluate every compared candidate against the same suite. The existing mutation lab provides a smaller local way to challenge a check before trusting its passes.

A.8.2 Make repair repeatable and acceptance explicit#

Keep a small repair loop near the changed behavior, then run the relevant integration and user-flow checks before acceptance. Control clocks, network responses, and starting state where they affect reproducibility. Record generated-input seeds so a failure can be replayed; deliberately vary seeds across additional runs to explore more cases. Preserve reduced failing inputs as regression cases when they protect a meaningful boundary.

Keep the first failure and investigate whether it belongs to the product, test, or environment. A later passing retry alone does not explain an intermittent failure. Report unresolved instability and the acceptance claim it limits. Useful failure output identifies the expected and observed behavior, the reproducing input, and relevant runtime evidence.

Review the final change for ownership, dependency direction, and maintenance consequences beyond the assertions. Record which revision and environment were checked, the results, and material omissions. Stop when the agreed evidence is sufficient for the change's consequences; add another check when it resolves a concrete uncertainty.

A.8.3 Compare repair evidence#

Changing the evidence supplied to a repair agent is an intervention in the loop. Start each trial from the same failing revision and synthetic data, with the same task, model, tools, limits, and independently specified acceptance checks. Vary only the additional diagnostic packet: none, raw telemetry, or a brief derived from that same telemetry. Keep the raw source available for auditing the brief, and record evidence collected during each repair so an arm does not silently acquire another arm's input.

Count instrumentation, collection, brief preparation, unsuccessful repairs, and human correction alongside accepted outcomes. Check that instrumentation preserves the observed behavior and that the repaired application passes without diagnostic instrumentation. A shorter brief can omit the decisive observation; a plausible diagnosis remains a hypothesis until the behavior is checked.

The optional telemetry-guided repair activity supplies an invented stale-list scenario and a bounded comparison protocol. Use it on paper to design the investigation or run fresh local trials on an application you control. One trial per condition is exploratory evidence; broader claims need repeated trials and varied failures under the uncertainty guidance.

A.9 Report uncertainty, not only point estimates#

An evaluation observes a finite set of tasks under declared conditions. A difference in average score or acceptance rate is therefore an estimate, not a stable ranking of two models or workflows. In a systematic review of 445 new or substantially modified LLM benchmark papers, only 16.0% used uncertainty estimates or statistical tests to compare results (Bean et al., 2025). Earlier work on NLP evaluations likewise found that small test sets can lack the power to distinguish plausible improvements and that effects detected by underpowered studies can be exaggerated (Card et al., 2020).

When possible, run both candidates on the same task IDs and predeclare how repeated calls become one task-level result. Repeated calls reveal variation within a task; they do not turn one task into many independent tasks. If many tasks are variants of one template, repository, or issue family, disclose that dependence and narrow the claim.

When a protocol permits k trials per task, name the reliability question before selecting a summary. Pass@k asks whether at least one of the k trials succeeds and is useful when several attempts are genuinely available and one accepted solution is enough. Pass^k asks whether all k trials succeed and is useful when users need the behavior to be dependable every time (Grace et al., 2026). More attempts make pass@k easier to satisfy and pass^k harder, so the same workflow can look increasingly capable by one measure and increasingly unreliable by the other. Predeclare k, budgets, stopping rules, and whether failed attempts remain in latency and cost totals; do not use pass@k to imply first-attempt reliability.

Report:

  • the independent task count and task source;
  • one predeclared primary outcome;
  • the observed paired difference;
  • a suitable paired interval and its method, when a course-provided or otherwise justified method fits the outcome.

Before inspecting results, define the signed difference and the practical boundary or bounds that would change the decision. One conservative, predeclared rule is to require a suitable interval wholly beyond the practical boundary before claiming the required effect. Under that rule, overlap leaves the effect claim inconclusive, while statistical separation can still describe a difference too small to matter. An interval containing zero is not proof that the candidates are equal.

An engineering action also depends on consequences, cost, and reversibility. A cheap reversible trial with monitoring and a rollback condition may be justified while the effect remains uncertain; an expensive irreversible migration may warrant more evidence. State the decision rule and its rationale separately from the claim about the measured effect.

Treat other metrics and unplanned slices as descriptive or exploratory unless their joint decision rule and multiple-comparison treatment were declared in advance. When no defensible interval is available, show the raw paired outcomes, label the comparison exploratory, and restrict the claim to those tasks. An interval describes uncertainty conditional on the sample and scoring process; it cannot repair unrepresentative tasks, judge bias or contamination, or account for a change to the agent or evaluation harness.

Paired comparison check. For one predeclared binary acceptance result per independent task, count tasks accepted by both candidates, only the first, only the second, or neither. Use an instructor-supplied interval or a justified paired method to interpret those counts against the predeclared decision boundary; do not compare two separate error bars. The single representative project task above remains a case comparison; use this check with a multi-task project cohort or instructor-provided data.

A.10 Qualify the served configuration#

A model name or local endpoint is not a capability result. Bind an accepted outcome to the configuration that served it: resolved model revision or digest and quantization, runtime version, allocated context, prompt and tool scaffold, and relevant hardware and accelerator residency or CPU/GPU offload where exposed. When an approved service does not expose a field, mark it unavailable and name the evidence source or service operator rather than estimating it.

In a study of 9.7K test examples across long-input and long-form-output benchmarks, quantization effects varied by method, model, task, context length, and language (Mekala et al., 2025). The study used vLLM on datacenter GPUs and did not test coding agents, Ollama, or consumer-device deployments, so one quantized build should not inherit another build's score. Hardware constrains which configurations fit and their latency, throughput, and resource use; it does not establish task capability.

Keep capability, hardware fit, and data location as separate claims. Running inference locally identifies where model computation occurs, not the data boundary of tools, telemetry, storage, or network calls. A fallback can select a different model or configuration and change the capability, latency, cost, and data boundary. Ollama, for example, exposes an installed model's digest and quantization and a running model's allocated context and server-reported model VRAM bytes. It can also forward cloud-model requests while callers continue using its local API, so localhost alone does not prove local execution (cloud models).

Assign a stable identifier to each served configuration in the evaluation. Record its exposed model, runtime, context, scaffold, tools, hardware boundary, outbound paths, and fallback policy, then link every attempt and acceptance result to that identifier. For an opaque departmental service, use operator- or instructor-supplied metadata; otherwise mark unavailable fields and name the evidence source. Qualify the configuration against a predeclared representative task and deterministic check, and treat the result as evidence for that task rather than a general ranking of local and hosted models.

Context selection and output reduction are also configuration decisions. Long tool outputs and interaction histories consume context and can increase latency or cost, which has motivated command-aware filters such as rtk and model-generated summaries. A shorter agent-facing view does not by itself establish a lower total bill or an unchanged task outcome. In a SWE-agent workshop study, observation masking and model-generated summaries reduced cost in most of five tested model configurations, but its paired analysis found that both significantly lowered solve rate in one reasoning-model configuration (Lindenbauer et al., 2025). On controlled long-horizon tool-use benchmarks, ACON reduced peak input tokens while largely retaining task performance (Kang et al., 2026). Neither study validates every command filter; together they show that context reduction is a behavior change to test, not free efficiency.

Compact the working view; preserve the raw record

Treat compressed output as a working view, not as the evidence record. Keep raw output retrievable during the run and preserve the producer's exit status independently. For bounded qualification cases and consequential failures, retain the original command or query, working directory, separate output streams, pagination or continuation metadata, and explicit omission markers under a stable handle. Apply the underlying system's data-minimization, secret-handling, residency, access, and deletion rules; disable archival where those rules cannot be met. Prefer native structured output as a comparison baseline, make capture failures visible, and treat retrieved output as untrusted text rather than embedded instructions.

Qualify each important compression policy with the same task cohort and acceptance checks used for the uncompressed view. Include compression calls, raw retrievals, retries, and failed attempts when reporting total cost per accepted outcome. A reduction in shell-output bytes is only a component metric.

Tool-output preflight. Use instructor-supplied or synthetic non-sensitive raw artifacts for one passing command and one deliberately tricky output: an exit-zero warning, a failing diagnostic, or an unusual line in a repetitive log. Production and secret-bearing output are outside this exercise. If the policy claims to handle pagination, add one paginated artifact.

Before generating compact views, list the critical facts in each raw artifact and require every compact fixture to mark at least one explicit omission. Record the command and tool version, reduction-policy version, configuration, and content hash. Verify that source status, separate output-stream provenance, critical facts, omission markers, retrieval handle, and pagination metadata survive. Report critical facts retained over critical facts present, false or materially misleading facts, and any policy-safe fallback. Passing this preflight can reject an obviously lossy or unsafe policy; claims about cost or task effectiveness require paired end-to-end outcomes.

A.11 Define the cost denominator#

Identify the tested configuration and cost population before interpreting an efficiency claim. The following numbers are invented teaching data. Suppose four distinct tasks each receive one attempt: two accepted attempts cost $1 and $3, while two failed attempts cost $8 and $12. The total measured execution spend is $24, the acceptance rate is 2/4, and there are no retries or missing usage records. The table below derives three different summaries from this same cohort.

Table A.6. Different cost populations produce different answers from the same invented task cohort. #
Measure Calculation Question answered
Median successful-attempt cost Median of $1 and $3 = $2 What did a typical successful attempt cost?
Mean attempted-task cost $24 / 4 attempted tasks = $6 What did an attempted task cost on average, including failures?
Total spend per accepted task $24 / 2 accepted tasks = $12 What did all attempted work cost per accepted task?

Calling all three numbers "cost per task" hides the distinction. If retries are allowed, include their spend while counting each accepted task once; label missing costs rather than treating them as zero. Report human review and other unpriced resources separately when they are outside the measured execution spend.

When interpreting cache savings, distinguish the median cache-hit rate across runs from the fraction of all input tokens served from cache. A few long runs can dominate token-weighted usage, and a high cache-hit rate can coexist with expensive failures. Before transferring a ranking to your project, test representative tasks with repeated fresh trials and the model, gateway, and cache conditions you intend to use.

A.11.1 Check whether an optimization repays its energy cost#

A faster implementation does not establish an energy saving. Compare a correct current implementation with an accepted candidate under the same workload and declared measurement boundary. Measure the additional energy spent producing the change, including generation, profiling, tests, and failed candidates, as E_change. Let e_before and e_after be energy per execution with equivalent behavior and service quality. After N executions, the net saving is:

net saving = N × (e_before − e_after) − E_change

When e_before > e_after, break-even requires at least ceil(E_change / (e_before − e_after)) executions. With positive change energy and zero or negative per-execution savings, there is no finite energy break-even.

For an invented example, suppose generation and rejected candidates use 18,000 J, while profiling and validation use another 6,000 J. An accepted change reduces energy from 1.2 J to 0.9 J per execution, saving 0.3 J each time. Recovering the 24,000 J investment requires 80,000 executions. If the implementation will run only 20,000 times before replacement, it saves 6,000 J in operation but increases total energy by 18,000 J. This arithmetic assumes stable savings and no additional maintenance or infrastructure energy; include those costs when they differ between alternatives.

Emami et al. (2026) measure generation and execution energy for iterative code optimization on a Python benchmark. Their reference programs are efficient competitive-programming solutions, and their refinement loop does not select candidates by execution energy. The study motivates measuring both costs; it does not predict the return from optimizing a particular production service. Use the expected lifetime workload and a plausible range of measured savings to judge your own change, and report unavailable energy measurements rather than substituting token counts or elapsed time.

A.12 From evaluation to routing#

A lower per-request price does not necessarily make a model cheaper for a task. An inexpensive attempt that fails may have to be repeated with a model that succeeds more often on that task, while still consuming context, validation, and review time. Conversely, always selecting the strongest available model can waste capacity when a smaller model can reliably clear the same acceptance bar. The useful comparison is therefore cost per accepted outcome, not token price or first-attempt cost.

Research distinguishes routing, which selects one model before it produces an answer, from cascading, which begins with one model and escalates when evidence indicates that its answer is insufficient. Both depend on an accurate quality estimate; Dekoninck et al. (2025) identify estimator quality as a critical condition for improving the cost-performance trade-off. In their SWE-Bench experiment, the post-generation estimator could use the task's ground-truth test outcomes. The result therefore shows what a strong verifier can enable, not how well the approach works for tasks without one.

When access rules permit both models—or when instructor-provided outputs are available—choose two models with different measured cost and task-success profiles within the same declared agent harness and assemble a small set of representative tasks. Keep near-duplicates and tasks from the same module in the same partition, using one partition to define the policy and reserving the other for held-out evaluation.

Define two evidence layers before running the comparison. The policy-visible gate may use deployable signals such as public tests, builds, or static checks to decide whether a cascade should escalate. The final adjudicator uses held-out tests or blinded human review that the policy cannot see; only this layer decides whether the final outcome is accepted. A policy that stops after passing its visible gate but fails final adjudication is a false acceptance.

On the first partition, define and freeze either a routing rule that uses only facts available before generation or a cheap-first cascade that uses the policy-visible gate after its first result. Freeze the task features, thresholds, prompts, versions of the model and agent harness, budgets, retry rules, and exclusion rules before inspecting the held-out results. Compare the policy with always using either model and, when the task set permits it, random selection at the same stronger-model call rate. The random baseline tests whether the policy selects useful cases rather than benefiting only from its mixture of models.

Report accepted outcomes over all attempted tasks, stronger-model calls, end-to-end latency, human correction, and total cost per accepted outcome. Count timeouts, tool failures, missing patches, and budget exits as attempted failures. Include policy decisions, quality estimation, rejected attempts, fallbacks, repeated context, validation, review, and rework in the cost; report unpriced resources separately. If both model arms are run to reveal counterfactual mistakes, keep that one-time evaluation expense separate from each policy's simulated operating cost. That comparison can expose false acceptances and unnecessary escalations; without both arms, label the missing counterfactuals as unknown rather than inferring them from an average.

The result supports a local decision only if the policy meets a predeclared quality and risk threshold while improving the chosen cost or latency measure. After using held-out results to change the policy, evaluate again on fresh tasks. High-risk work can bypass the router or retain mandatory review, and the policy should be recalibrated when models, prices, agent harnesses, or the task distribution change.

A.12.1 Check confidence before routing#

A closed choice set can simplify output validation. An allowed label can still be wrong, and the correct outcome may be missing from the choice set.

Confidence calibration asks whether reported probabilities agree with the frequency of correct predictions over relevant cases (Guo et al., 2017). A score of 0.9 needs evidence from comparable predictions before it can guide a routing threshold. Confidence about a classification also concerns a different outcome from whether the resulting patch passes acceptance or an action is authorized. Keep mandatory checks and permissions outside the learned score.

To test this distinction, give your agent the confidence starting point, or examine the invented classification batch yourself.

A.13 Compressed evidence and repeated review#

Use compact views to direct attention and repeated reviews to search for missed defects. Keep a common acceptance protocol:

  1. Preserve the full artifact and independent checks, and inspect omitted material at consequential decisions.
  2. Confirm each finding through a reproduction, reachable path, or violated invariant before judging its impact.
  3. Record triage and repair effort, and look for a common cause when several findings recur.
  4. Stop against a declared risk threshold; decompose a change that remains too large or coupled to understand.

The cases below show why these controls matter.

A reading diff routes attention

Large changes may repeat the same migration across call sites or include generated and mechanically forced edits. Compact views can reduce repetition while hiding distinctions needed for acceptance. An author-run reading-diff analysis documented hidden test outcomes and an obscured old/new default change, rather than measuring reviewer effectiveness. A mutation-based study of code summaries also found descriptions of familiar algorithmic intent that missed changed behavior; it did not evaluate diffs or human reviewers (Khatib et al., 2026).

When findings become inventory

Continued review creates reports that still require verification and a repair decision. A live study at Mozilla and Ubisoft found some generated comments useful, but reviewers directly accepted only a minority (Olewicki et al., 2026). It measured adoption rather than defect-detection accuracy. Count verification and repair effort when judging whether the review helped.

A.13.1 Review-evidence exercise#

Choose one bounded representative change. Before opening a compact or agent-produced review view, predeclare two or three behavior or architecture questions. Record provisional answers after that view, then inspect the complete artifact and relevant checks and record every correction, omitted or misleading critical detail, and the time spent in each stage. For agent findings, record candidate, independently confirmed, duplicate, and rejected findings together with triage and repair time. Apply the acceptance protocol above when deciding which findings to repair and when to stop.

A.13.2 Optional exercise — Challenge an explanation#

Predict what a small event recorder does when a worker restarts, then check an agent's explanation against the code. Give your agent the explanation starting point, or work through the fixture yourself.

A.14 Repeated review is measurement, not proof#

Repeating an agent review can reveal variance, but agreement between runs is not independent validation. A repeated security-review study found more stable recurrence for findings that matched its static-analysis reference than for unmatched findings (Tal et al., 2026). That reference agreement did not establish ground-truth accuracy.

Treat each model review as one sample. Deduplicate repeated findings, record recurrence separately from deterministic-reference overlap, and preserve rare findings for case-level adjudication rather than automatically discarding them. Consensus can repeat a shared blind spot, while a low-frequency report can still identify a real defect. Behavioral tests, invariants, deterministic analysis, and accountable review remain the acceptance evidence.

Repeated review also consumes the same scarce integration capacity that bounds the fleet. If you repeat an agent review, record the fixed code, prompt, agent harness, and model configuration; the number of runs; unique and recurring findings; reviewer time; and cost per accepted finding. More review samples are justified only when they improve accepted outcomes or allocate human attention better than a single review.

A.15 Research example: coordination and integration#

CodeCRDT's experiments on six small TypeScript and React tasks used a shared CRDT document that converged without textual merge failures, while preliminary inspection still found semantic conflicts (Pugachev, 2025). For example, interface and data code can use different event names in a document that merges cleanly. After concurrent work, test the integrated user journey against the settled contract.

Cemri et al. (2025) provide a broader diagnosis catalog: over 1,600 annotated traces from seven multi-agent frameworks and a taxonomy of 14 failure modes involving system design, inter-agent misalignment, and verification. Expert analysis of 150 traces informed the taxonomy; annotation at larger scale also used an LLM judge. These categories describe the studied traces, not an industry-wide failure rate or proof that a role template repairs coordination. For one observed failure, connect a proposed category to the actual task, shared state, handoff, and acceptance check, then verify the repair through behavior.

For further reading, Cho & Lee (2026) examine coordination through shared files, claims, and messages; Ogenrwot & Businge (2026) examine textual merge conflicts in agent-authored pull requests. Read each study's task selection, acceptance checks, and coordination boundary before transferring its conclusions to a fleet design.

A.16 Field note — Let repeated failures shape the first evals#

My practical threshold for codifying an eval is when I begin to see a pattern and trust my judgment enough to describe it. I do not yet have enough experience to claim a general method for designing strong eval suites, however. So far, I have mostly allowed development models to propose the evaluators, then complained about obvious weaknesses I found while reviewing their code.

Slideotter provides the clearest example of where this still became useful. The repository preserves the resulting generation fuzz harness. I asked a stronger development model to mutate presentation scenarios and exercise the smaller local models through that harness. The mutations explored combinations and failure cases I would not have devised on my own.

Some outputs failed in surprising ways. Those failures became new regression scenarios, stronger validators, and improvements to the surrounding prompts, schemas, and generation scaffolding. The resulting loop helped me extract more useful behavior from weaker local models.

This was closer to mutation-based probing than an independent evaluation of presentation quality. The development model could introduce its own blind spots, and model-generated evaluators could reward the same assumptions that shaped the implementation. Structural and rendering checks could establish that a presentation was valid under the tested scenarios; they could not establish that it communicated the topic well.

I therefore treat generated evals as a way to discover and preserve recurring failure patterns, not as an automatic source of ground truth. Editorial judgment remains necessary until repeated observations become clear enough to encode and the evaluator itself has been challenged against known successes and failures.

A.17 References#

  1. Grace, M., Hadfield, J., Olivares, R., & de Jonghe, J. (2026, January). Demystifying Evals for AI Agents. Anthropic Engineering. https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents
  2. Terminal-Bench Contributors. (2026). Terminal-Bench 2.1. GitHub repository. https://github.com/harbor-framework/terminal-bench-2-1
  3. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., & Stoica, I. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv Preprint arXiv:2306.05685. https://doi.org/10.48550/arXiv.2306.05685
  4. Dekoninck, J., Baader, M., & Vechev, M. (2025). A Unified Approach to Routing and Cascading for LLMs. Proceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, 267, 12987–13010. https://proceedings.mlr.press/v267/dekoninck25a.html
  5. Pugachev, S. (2025). CodeCRDT: Observation-Driven Coordination for Multi-Agent LLM Code Generation. arXiv Preprint arXiv:2510.18893. https://doi.org/10.48550/arXiv.2510.18893
  6. Cho, S., & Lee, D. (2026). AgentRoom: Concurrent Multi-Agent Coding in a CRDT-Backed Shared Workspace. arXiv Preprint arXiv:2608.23740. https://doi.org/10.48550/arXiv.2608.23740
  7. Ogenrwot, D., & Businge, J. (2026). AgenticFlict: A Large-Scale Dataset of Merge Conflicts in AI Coding Agent Pull Requests on GitHub. Proceedings of the 3rd ACM International Conference on AI-Powered Software. https://doi.org/10.1145/3805760.3814923
  8. Tal, L., Kloos, J., Rudich, A., Thoemmes, S., & Nair, M. (2026). Snyk VulnBench JS 1.0: Can LLMs Find the Same Bugs Twice? arXiv Preprint arXiv:2606.15762. https://doi.org/10.48550/arXiv.2606.15762
  9. Olewicki, D., Da Silva, L. M. P., Ben Sghaier, O., Mujahid, S., Amini, A., Mah, B., Castelluccio, M., Habchi, S., Khomh, F., & Adams, B. (2026). Impact of an LLM-Based Review Assistant in Practice: A Mixed Open-/Closed-Source Case Study. IEEE Transactions on Software Engineering, 1–12. https://doi.org/10.1109/TSE.2026.3663093
  10. Khatib, L., Pu, M., Vasilescu, B., & Nagappan, M. (2026). Using Mutation-Analysis to Examine an LLM’s Ability to Summarize Code. https://doi.org/10.48550/arXiv.2602.17838
  11. Bean, A. M., Kearns, R. O., Romanou, A., Hafner, F. S., Mayne, H., Batzner, J., Foroutan Eghlidi, N., Schmitz, C., Korgul, K., Batra, H., Deb, O., Beharry, E., Emde, C., Foster, T., Gausen, A., Grandury, M., Han, S., Hofmann, V., Ibrahim, L., … Mahdi, A. (2025). Measuring what Matters: Construct Validity in Large Language Model Benchmarks. Advances in Neural Information Processing Systems, 38. https://doi.org/10.52202/085713-0590
  12. Card, D., Henderson, P., Khandelwal, U., Jia, R., Mahowald, K., & Jurafsky, D. (2020). With Little Power Comes Great Responsibility. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 9263–9274. https://doi.org/10.18653/v1/2020.emnlp-main.745
  13. Kang, M., Chen, W.-N., Han, D., Inan, H. A., Wutschitz, L., Chen, Y., Sim, R., & Rajmohan, S. (2026). ACON: Optimizing Context Compression for Long-horizon LLM Agents. Proceedings of the 43rd International Conference on Machine Learning, Proceedings of Machine Learning Research, 306. https://arxiv.org/abs/2510.00615
  14. Lindenbauer, T., Slinko, I., Felder, L., Bogomolov, E., & Zharov, Y. (2025). The Complexity Trap: Simple Observation Masking Is as Efficient as LLM Summarization for Agent Context Management. Fourth Deep Learning for Code Workshop (DL4C) at NeurIPS 2025. https://doi.org/10.48550/arXiv.2508.21433
  15. Mekala, A., Atmakuru, A., Song, Y., Karpinska, M., & Iyyer, M. (2025). Does quantization affect models’ performance on long-context tasks? Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 9422–9470. https://doi.org/10.18653/v1/2025.emnlp-main.479
  16. Orlanski, G., Roy, D., Yun, A., Shin, C., Gu, A., Ge, A., Adila, D., Roberts, N., Sala, F., & Albarghouthi, A. (2026). SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks. arXiv preprint arXiv:2603.24755. https://doi.org/10.48550/arXiv.2603.24755
  17. Yao, S., Wang, H., Sen, K., Chen, S., Ray, B., & Song, D. (2026, October). TestJack: Should You Trust the Results in Coding Benchmarks? Agentic Coding Benchmarks Auditing via Evaluator Evolution. arXiv preprint arXiv:2610.10619. https://doi.org/10.48550/arXiv.2610.10619
  18. Lulla, J. L., Nersesyan, V., Mohsenimofidi, S., Treude, C., & Baltes, S. (2026, August). Loop Engineering: Building Blocks, Adoption, and Impact. arXiv preprint arXiv:2608.21884. https://doi.org/10.48550/arXiv.2608.21884
  19. Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., Gupta, S., Majumder, B. P., Hermann, K., Welleck, S., Yazdanbakhsh, A., & Clark, P. (2023). Self-Refine: Iterative Refinement with Self-Feedback. Advances in Neural Information Processing Systems, 36. https://doi.org/10.52202/075280-2019
  20. Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., & Yao, S. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. Advances in Neural Information Processing Systems, 36. https://doi.org/10.52202/075280-0377
  21. Olausson, T. X., Inala, J. P., Wang, C., Gao, J., & Solar-Lezama, A. (2024). Is Self-Repair a Silver Bullet for Code Generation? International Conference on Learning Representations. https://arxiv.org/abs/2306.09896v5
  22. Stewart, L. (2026, September). Agentic Alienation. Loren Stewart’s blog. https://www.lorenstew.art/blog/agentic-alienation
  23. Mollick, E. (2026, September). The Overhang. One Useful Thing. https://www.oneusefulthing.org/p/the-overhang
  24. Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On Calibration of Modern Neural Networks. Proceedings of the 34th International Conference on Machine Learning, Proceedings of Machine Learning Research, 70, 1321–1330. https://proceedings.mlr.press/v70/guo17a.html
  25. Lemire, D. (2026, October). Ephemeral Testing. Daniel Lemire’s blog. https://lemire.me/blog/2026/10/05/ephemeral-testing/
  26. Ahmad, N., Su, R., Esposito, M., Janes, A., Lenarduzzi, V., & Taibi, D. (2026). Architectural Degradation: How to Measure and to Remediate. arXiv Preprint arXiv:2610.01611. https://arxiv.org/abs/2610.01611
  27. Shen, J. H., & Tamkin, A. (2026, February). How AI Impacts Skill Formation. arXiv preprint arXiv:2601.20245. https://doi.org/10.48550/arXiv.2601.20245
  28. Bastani, H., Bastani, O., Sungu, A., Ge, H., Kabakcı, Ö., & Mariman, R. (2025). Generative AI without Guardrails Can Harm Learning: Evidence from High School Mathematics. Proceedings of the National Academy of Sciences, 122(26), e2422633122. https://doi.org/10.1073/pnas.2422633122
  29. Cemri, M., Pan, M. Z., Yang, S., Agrawal, L. A., Chopra, B., Tiwari, R., Keutzer, K., Parameswaran, A., Klein, D., Ramchandran, K., Zaharia, M. A., Gonzalez, J. E., & Stoica, I. (2025). Why Do Multi-Agent LLM Systems Fail? Advances in Neural Information Processing Systems, 38. https://doi.org/10.52202/085713-4082
  30. Appleton, M. (2026, September). Planning with Agents: Divided Worlds, Boundary Objects, and Thicker Interfaces. https://maggieappleton.com/planning-agents
  31. Niska, C. (2026, October). The Outer Loop. https://crisu.me/blog/the-outer-loop
  32. Contractor, Z., & Reyes, G. (2026, October). Experimental Evidence on the Learning Impact of Generative AI. arXiv preprint arXiv:2607.08849. https://doi.org/10.48550/arXiv.2607.08849
  33. Emami, S., Weigell, B., Mousavi, M. R., Jahangirova, G., Araujo, H., & Bauer, B. (2026). Green Code or Costly Code? Evaluating the Energy Costs of Iterative LLM-Based Code Optimizations. https://doi.org/10.48550/arXiv.2610.32891