What the repair robots remembered

Intuitively, giving an AI agent memory should help it do a better job. If it has solved a problem before, why make it figure everything out again?

I wanted to test that assumption with a simple harness and evaluation suite. Something small enough to inspect, but with the kinds of decisions that come up in the agent workflows we build for clients at Jeavio: work out what’s happening, choose a tool, act on the result, and check whether the job is actually done. Earlier experience should sometimes help. Sometimes it should be ignored.

So I built a fictional repair shop. The agents are robot mechanics working through a queue of faulty machines. The interface has an 80s workshop theme (though I may have gone a little overboard with the design - everybody smoked in the 80s right?). Putting dodgy robots aside, the experiment underneath it is about memory, prompts, tool use and cost.

Original three-bay robot workshop in free-demo ready mode

Figure 1. The original three-bay workshop, shown in free-demo ready mode.

I wanted to check the sensitivity of the harness to the model, memory infrastructure and token budgets. I also put together three task sets of increasing complexity to test out my hypothesis.

The results? Well, honestly, still a work-in-progress! With Luna, memory saved work on some repairs but added tokens and left jobs unfinished. Changing the repair model altered that tradeoff again. The experiment continues to validate my hypothesis that agentic development requires an iterative and empirical approach!

What happens in the shop #

For the comparisons here, I moved from those initial task sets to an eight-job service suite. Every machine arrives with the same complaint: intermittent shutdown under load. The agent has to work out whether it has a calibration, cooling or cable problem, make any necessary repairs, and pass an independent final test. One machine is perfectly healthy, so doing more work isn’t always the right answer.

The simulation decides what each tool action does and whether the machine passes. The harness records what the agent tried, what happened and how many tokens it used; the agent doesn’t get to grade its own work. That gives me a way to distinguish a convincing explanation from a completed repair.

The sequence mixes familiar situations with cases where earlier experience could mislead the agent. A component returns with the same calibration but a new cable fault: remembering the calibration can save measurements, but replaying the old repair won’t fix the cable. A replacement component needs fresh measurements. I wanted to see whether the agent could recognize which part of an earlier repair still applied.

That is the connection to client workflows. The tools and business rules will be different, but an agent still has to decide whether an earlier answer applies to the case in front of it. The shop gives me a small, inspectable place to test that judgment. The full task descriptions and rules are on GitHub.

Three memory approaches #

I gave three agents different ways to handle experience from earlier jobs. I’ll use their labels, No memory, Rolling memory and Structured memory, throughout:

No memory starts each new job without the previous jobs’ observations.
Rolling memory keeps a bounded window of previous observations, without summarizing them.
Structured memory uses a separate memory-writing agent to condense the work log into records of past repairs, component facts and procedures.

All three retain the observations from the job they are currently doing. The No memory label means no history from earlier jobs, not forgetting the tool result that just came back. The logs are factual English descriptions of actions and results, rather than the model’s private reasoning.

The structured approach adds a step between finishing one repair and starting the next. It keeps working memory for the current job, then asks the writer to extract useful experience from the job’s work log when budget remains. Episodic records describe past repairs; semantic records hold facts such as a component’s calibration; procedural records describe how to do something. These are record types in one store, not three more agents.

Each agent attempts the same eight jobs in the same order across three fresh shifts. Memory resets between shifts, giving 24 attempts per memory approach, rather than 24 different cases. Within a model run, the model, reasoning setting and tools stay fixed, so I can compare the memory approaches without also changing the repair agent.

Three memory approaches in a robot repair-shop experiment. Each approach gets the same eight repair tickets across three fresh shifts, for 24 attempts. Within each model run, the model, reasoning setting and tools stay the same. All three agents keep observations from the current job. No memory discards observations from previous jobs. Rolling memory keeps recent observations as whole entries within a fixed limit. Structured memory uses a writer after each job to save past repairs, facts and procedures, with notes about when they apply. Token use and estimated cost are recorded separately from memory, and an independent test checks each repair. These observation logs are factual records, not private chain of thought.

Figure 2. The three memory approaches receive the same jobs and tools, and each retains its own current-job observations. They differ in what they retain and supply from earlier jobs.

I also need limits on how much work an agent can do. Each job gets up to 20 repair actions and an allowance of 50,000 input and output tokens, including the memory writer. The token allowance is checked between calls, so a final admitted call can take the total over that limit. A separate whole-run budget limits the cost of the experiment.

I measure completion, repair actions, tokens and estimated API cost separately. Fewer actions can mean that the agent found a shortcut, or simply that it stopped before finishing. And memory has a cost of its own: previous observations have to be included in later requests, while the structured approach also pays for a writer to maintain its records.

Useful memory went unused #

The earlier run with gpt-6-luna did not give me the result I’d expected. No memory passed 23 of 24 repairs; Rolling memory and Structured memory each passed 21. Both memory approaches completed fewer repairs than the baseline in that run.

The action logs made the problem more interesting. Structured memory took both calibration samples on every ticket, even when the correct calibration was already in the memory supplied to it. The information had survived storage and reached the agent, but the agent still repeated the measurements.

That gave me something more specific to investigate than whether memory was a good idea. The writer was producing overlapping records and describing useful procedures too narrowly. I also suspected the repair prompt wasn’t clear enough about when the agent could rely on earlier evidence. The model was part of this setup, but so were the instructions and the way I presented its memory.

I revised both sides. I asked the writer for shorter facts with clearer descriptions of where they applied, and told the repair agent to consult memory, reuse supported facts and check for exceptions. I also tightened the task rules and added counterexamples, including a familiar component with a new cable fault. The agent needed to reuse useful knowledge without assuming that the entire previous repair would work again.

After the offline tests passed, I ran the suite again. Because the tasks and prompts changed together, the result can’t tell me how much improvement came from the memory instruction alone. The earlier report preserves the first setup and its results separately.

Fewer repair actions did not mean fewer tokens #

With the updated setup, Structured memory started behaving more like I’d intended. It took 18 calibration samples across 24 attempts, compared with 48 in its earlier run. It reused familiar calibrations but still measured the replacements and the new component.

On the familiar component with the added cable fault, it finished in eight actions in every shift, against ten or eleven for No memory. It skipped the unnecessary measurements and still fixed the new fault. These were completed repairs, so there was useful work being saved.

The improved reuse still wasn’t enough to match No memory on completion across the whole suite. No memory finished all 24 repairs, while Rolling memory finished 21 and Structured memory finished 22. No memory also used the fewest tokens.

Tuned Luna results comparing the memory approaches No memory, Rolling memory and Structured memory. Out of 24 attempts each, they completed 24, 21 and 22 repairs respectively, verified by the independent test. Total tokens were 524,517, 768,076 and 653,012. Structured memory includes 93,646 tokens used to write memory. Repair actions totaled 248, 211 and 205, including unfinished jobs. Across all approaches, 72 jobs were attempted, 67 passed and 5 were unfinished. All five unfinished compound repairs reached their job token limit. Fewer actions can reflect unfinished work rather than greater efficiency. The run used 1,945,605 tokens at an estimated cost of $0.2333469. This is a small experiment, not a general ranking. The measured version is service-v2, repair-v3 and writer-v2, frozen run f20daf0, report 2448a7a.

Figure 3. Results from the tuned Luna run. Actions and tokens include unfinished work; tokens also include the memory writer.

All five unfinished attempts were compound repairs, where the agent had several faults to diagnose and fix. They exhausted their token allowance before passing the final test. The extra history was included in successive model calls, so avoiding a few repair actions did not necessarily reduce the total amount of text processed.

This is why I wanted separate measures. The logs showed genuine reuse on particular repairs, while the totals showed that it wasn’t enough to make either memory approach cheaper or more reliable across this suite. Structured memory used fewer tokens than in its earlier run, but still more than No memory, including the writer. The full report has the per-attempt breakdown.

Changing the repair model #

At this point, the agent was reusing memory, but still struggling to finish some jobs within the allowance. Would a more capable repair model make better use of the same setup? I swapped gpt-6-luna for gpt-6-sol, keeping the tuned tasks, prompts, medium reasoning and per-job allowances unchanged. The memory writer stayed on Luna.

With Sol repairing, Structured memory passed all 24 jobs, matching No memory. It used 30 fewer repair actions at that same completion rate. That looks more like what I’d expected! But compared with No memory on Sol, it still used 38.8% more tokens and cost an estimated 17.8% more.

Sol results comparing two memory approaches: No memory and Structured memory. Both completed all 24 repairs, verified by the independent test. No memory took 234 repair actions and Structured memory took 204, a reduction of 30. Total tokens were 470,232 and 652,515, so Structured memory used 38.8 percent more. Estimated model costs were $1.048392 and $1.235156, a 17.8 percent increase. Structured memory includes 557,819 repair tokens and 94,696 tokens used by the fixed Luna model to write memory. The repair model was Sol at medium reasoning. This is one synthetic suite, not a general ranking. Rolling memory is omitted from this focused chart; the full report includes its three job-token failures and earlier spending interruption. Report 89e91d7, measured source f18a9b8.

Figure 4. The Sol comparison focuses on the two agents that completed every repair: No memory and Structured memory. Totals include the Luna memory writer.

The compound repairs explain the change in completion. With Sol, Structured memory passed all three in 14 actions each; with Luna, it had passed only one. Sol gathered the evidence and completed the repairs in one power-off period before restoring power and testing. Those final calls took the jobs just over the nominal 50,000-token allowance, which the between-call checks permit.

The higher completion rate also came with a different bill. For Structured memory, changing the repair model took the estimated total cost from about $0.081 to $1.235, with almost no change in total tokens. The model was reaching a better outcome with a similar amount of text, but that text was much more expensive to process.

There was a complication in the Sol run: it initially hit the fifty-cent whole-run spending ceiling. I raised that ceiling to $10 and continued the untouched jobs, leaving the interrupted Rolling memory repair in the results. That agent completed 20 of 24 repairs: three exhausted the job allowance and one was interrupted by the spending limit. Four later jobs inherited the interrupted repair’s incomplete history. No memory and Structured memory were unaffected; the report and matched comparisons include the interruption and the comparisons that exclude affected history.

Changing the repair model improved completion for Structured memory in this run. But these are three repetitions of one synthetic sequence, and the improved memory reuse had already appeared after the earlier task and prompt changes. The Sol result doesn’t establish that Luna alone caused the original problem.

Repeating the exact same job #

The service suite deliberately asks the agent to distinguish a useful repeat from a changed situation. I also wanted a simpler check: could Luna reuse what it had learned when the next job was exactly the same?

For that test, I gave each memory approach three fresh pairs of identical repairs. Each pair began with no history. The machine reset between its two jobs, so the second repair still had to be done and independently tested; only memory from the first job could persist. The tuned prompts and tools stayed the same.

All 18 repairs passed. No memory took nine actions on both jobs in every pair. Rolling memory and Structured memory each went from nine actions to seven, skipping the two calibration measurements on every repeat while still checking and repairing the current machine.

I checked what the agents actually received before the repeat. The entire first-job log fitted into the rolling window, so no earlier observations had been lost. Structured memory supplied a much shorter calibration record. Both approaches then acted consistently with using that earlier evidence.

Even in this simpler case, fewer actions didn’t mean lower overall cost. Across the complete learning-and-repeat pairs, Rolling memory used 4.2% more tokens than No memory; Structured memory used 7.8% more, including its writer. Their estimated costs were 5.2% and 20.8% higher respectively. These totals include learning on the first job and the repeat, not just the second repair. The structured approach also paid for a memory write after the second repair, even though this short test had no third job to benefit from it.

This gives me a narrow but useful answer: Luna could reuse the supplied experience on this exact repair, with the tuned setup. It doesn’t rank the approaches for other work. It also leaves a practical question about whether more repetitions would recover the extra cost of memory, and if so, how many. The exact-repeat report and task description have the details.

Where the development effort goes #

I started with a fairly obvious-sounding assumption: give the agent memory and it should do better. Well, better at what? Remembering something useful, making fewer tool calls, finishing the job and spending fewer tokens didn’t all improve together.

That’s the architectural decision I want data for. If a tool call is expensive or slow, avoiding it may be worth extra model tokens. If completing the job within a fixed budget is what matters, a memory approach that saves actions but runs out of tokens is a poor fit. The useful comparison depends on the work we need the agent to do and what that work costs.

There are still changes I want to test separately: the repair prompt, the memory writer, and the amount of context supplied. I also want to try cases the setup hasn’t been tuned around. The experiments so far tell me what happened in this small shop; they give me a way to ask better questions before choosing an architecture for a client workflow.

And this brings me back to the 80/20 inversion. Building the first version gave me something to try. Most of the interesting work since then has been figuring out what to measure, looking at what the robots actually did, changing the setup and checking again. That’s how I want us making decisions on client agent projects.