Building Personal Harnesses

How do you decide on technology when things move this fast? For me, it used to start with a book.

Eleven years ago, I spent two weeks immersed in a book about Docker because releases kept going wrong and ruining my weekends.

My review ended with two questions: “Will Docker solve my release woes? Is it actually ready to be deployed in a corporate setting?”

Swap Docker for agents, and these are the questions leaders face this year.

Investors, the board, and customers all want to know what you’re building with AI, and the proposals arrive with more hype and bigger budgets.

If you make an application agentic, or put millions of your 2027 budget into an AI transformation, what are you signing up for?

I still read, I still talk to engineers, and I still work out where new technology fits. Now I also build small experiments, which Claude Code and Codex have made quick and cheap.

I call them personal harnesses: small programs that wrap AI models around one task, so I can change one thing and watch what happens. I build them in areas I already understand, because then I can tell when the answer is wrong.

Every agent project hides a few decisions: which model runs it, what it remembers, how it finds information, and which trade-offs you can live with.

In one experiment, four AI agents negotiated data center deals with a simulated county. I used the same prompts, tools and configuration across four models. All four got close to the best available deal. Their effective input-token costs varied far more than the price sheets suggested, and the cost ranking changed.

In another, I gave the agents in a fictional robot repair shop a memory of earlier jobs. Memory sounds like an obvious upgrade. But in one run, structured memory used about 24% more tokens, including the memory writer, and finished fewer repairs: 22 of 24, against 24 of 24 without it. The agent with structured memory took fewer repair actions, and two of its jobs hit their token limit before finishing.

A third harness searches 176 of my blog posts, going back to 2010. When I asked it for “the book I read to stop my releases breaking,” keyword and hybrid search both found the Docker review. With semantic search, it couldn’t identify the book from the retrieved excerpts, and said so.

AskRushi comparison for the question “the book I read to stop my releases breaking.” Semantic search does not identify a book; keyword and hybrid both identify The Docker Book by James Turnbull. Cropped screenshots show the revealed methods and their actual answers. — One saved comparison, with the method labels revealed. Keyword and hybrid found the Docker review; the answer using semantic search said the supplied excerpts did not identify the book. This result does not rank the methods for other questions.

View the full-size comparison.

These experiments were small. They gave me better questions for the next team that brings me an agent project, and for clients working out their 2027 AI strategy.

Before I sign off on an agentic project, or advise a client to, I want to have built a toy version of its riskiest part. Then I throw the toy away and keep the questions.