Challenge · Optional exploration
Probe an API with temporary clients
A generated CSV client reports success but exports two rows. Design independent checks for a fixed paginated event feed.
Think it through with your agent
Give your agent this URL. It contains the scenario and guidance to ask one question at a time. Start with your own prediction or explanation, then check it against the book.
Use with your task. Repeat the evidence collection. Keep discoveries in your existing evidence or learning notes.
Follow the argument. Probe an API with temporary clients.
Read the exercise
Scenario#
Invented local fixture.
An event-query API serves synthetic records through GET /events?workspace=<id> and an optional cursor parameter.
Its declared contract returns items and nextCursor; null marks the last page.
Workspace W1 contains events E1, E2, and E3 in that order, and W2 contains E4 and E5.
The actor may read only W1; a W2 request must return 403 without event data.
Pages contain at most two events, so the declared W1 sequence is E1/E2 with a cursor, followed by E3 with nextCursor: null.
A generated CSV client reports success but exports two rows. You have not inspected its requests or output IDs. A second client is intended to count all authorized events for a workspace summary. The provider revision and public contract remain fixed while clients are built and investigated.
Investigate#
- Predict what the CSV client's successful exit and two-row output establish. Which evidence would locate a problem in the provider, client, documentation, or acceptance check?
- Before generating another client, define independent checks for the exported IDs, ordering, duplicates, summary count, and forbidden workspace access. Derive expected results from the declared fixture, and specify a known-correct request sequence or reference client as a control.
- Give the public contract to an agent and ask it to build one temporary CSV or summary client in a disposable local directory. Use synthetic records, a fixed time budget, and read-only access to the provider's source; record its configuration and any request or attempt to change the provider. You can instead use the paper scenario to design the probe, labelling its outcomes as predictions.
- Compare the generated client's behavior with the control and independent checks. Reduce any failure to a request, input, and observation, then distinguish a provider defect, consumer bug, environment fault, confusing documentation, or incorrect expectation. If the evidence cannot distinguish them, retain that uncertainty and name the next check.
- Identify what can be discarded and what must survive. Preserve the contract/provider revision, relevant synthetic inputs, request trace, minimal reproduction, and confirmed regression case before removing temporary scaffolding.
Record useful integration outcomes, failures, investigation time, and attempts within the chosen budget. Do not count every failed generated client as a provider defect or every passing client as complete contract coverage. Keep the findings in existing evaluation notes; this activity adds no required project feature or submission.
Further reading#
Daniel Lemire's ephemeral-testing proposal uses disposable downstream applications to probe a maintained component. Appendix A connects the method to independent checks and retained regression evidence.
Show worked reasoning
A successful exit establishes that the process completed under its own success rule.
It does not establish that the CSV meets the declared API and export requirements.
For this fixture, a complete authorized export contains E1, E2, and E3 once each in order, and the summary count is three.
A W2 request must return 403 without revealing E4 or E5.
The two-row observation alone does not identify the faulty component.
Inspect the exported IDs and request trace, then compare them with a control that follows each returned cursor until nextCursor is null.
If the provider supplies the declared second page but the CSV client never requests it, the missing traversal belongs to the client.
If a correct control requests that page and the provider omits E3, the provider violates the fixture's declared contract.
If the CSV has all three events but a validator expects two, the check is wrong.
An unreachable local endpoint is an environment problem until evidence shows otherwise.
A contract that clearly names pagination can be implemented incorrectly; a contract that omits how to finish traversal may instead expose a documentation problem. Keep usability findings separate from confirmed functional defects. The control and expected IDs must be derived independently of the generated client's logic, or both can repeat its assumption. Client-side workspace filtering is also insufficient: the API must reject the unauthorized W2 read at its access boundary.
Keeping the provider fixed makes a consumer failure inspectable. Allowing the agent to silently patch the provider could make the final demonstration pass while hiding which contract or component changed. A confirmed provider defect can justify a separate repair after its reproduction is retained. Likewise, a repaired client should face the same independent checks rather than treating the agent's explanation as acceptance.
Discarding a probe application need not discard its useful evidence. Retain the minimal failing case, synthetic inputs, revision, and observed requests; turn a confirmed defect into a maintained regression check at the responsible boundary. One passing probe covers its exercised cases, while broader claims need additional consumers, inputs, and independently adjudicated failures.
This is optional, ungraded exploration. The course brief defines the required work.