Test the part that does the work

The reply was fine. The command runner was missing.

A coding agent gave a plausible reply in a smoke test. The same package failed when it had to read files or run a command: its command-host helper was missing. The first test had exercised the model while leaving the working part of the service untouched.

The repair included the helper in the package and checked it when the agent launched. On 1 October 2026, the follow-up test ran a repository command and compared the answer with the actual repository. That was a small test, but it crossed the boundary I depended on.

Choose the question before choosing the test

For a summariser, the quality of the text may be the result. For an agent that changes a record, the question is whether the right record changed. For one that starts a job, the question is whether the job finished. A single success flag cannot answer all three.

I find it useful to separate the reply, the tool action and the resulting work. This keeps a good explanation from covering a failed action. It also makes the failure easier to locate: was the input wrong, did the tool fail, or did another system show an older value?

A retry can create more work

A timeout leaves an awkward question: did the operation fail, or did the response fail to reach me? Sending the request again may create a duplicate. Waiting may leave the original work unfinished.

AWS describes this problem in its guidance on idempotent APIs. A caller supplies an identifier for the intended request so a retry can be recognised as the same operation. That identifier has to preserve intent; reusing it for a different action creates a different error.

A delayed view needs another response. Amazon EC2 documents cases where a newly created resource temporarily appears absent. Its guidance checks the resource state with increasing waits between attempts. Retrying the creation and waiting for the existing resource are different choices.

The extra check has a cost

Checking every layer adds time, API calls and more failure paths. For a short task, a committed record returned by the tool may be enough. For a high-cost change, I want to inspect the result where it will be used. Repeated low-cost work may justify sampling once the process is understood.

The choice depends on the cost of a missed error and how quickly it would be found. A check that merely repeats the model’s explanation buys little. A check of an independent record can answer a new question.

Make the test do useful work

I now start with the smallest task that uses the service’s essential capability: run a command, retrieve a cited document or change a test record and inspect it. Then I add the failure I am concerned about, such as a missing dependency, a delayed response or a repeated request.

The practical lesson is to test the working path early. A convincing answer is cheap evidence when the job depends on something else.