Where to invest in AI, and how to run it in production.

A companion to "Where to Invest in AI," the first webinar in our series with Andrei. The full recording is below.

Andrei opened the session by asking me the one piece of advice I wish more PE-backed companies were taking seriously about AI. I've been building AI products at Jeavio since 2017, so I think he expected something grand.

The honest answer is deflating: treat it like a normal technology.

We've spent fifty or sixty years learning how to fold new technology into our products and our companies, and almost all of those lessons still apply. The mistake I see most often is treating AI as a special case that suspends the old rules. It doesn't. So this is the companion piece to that session. If you were there, this is the map to keep. If you weren't, start here.

1. The frontier is jagged, and it keeps moving

What "jagged" means

Ethan Mollick, who writes about this more clearly than most, calls it the jagged frontier. The idea: AI capability isn't a clean line you can draw between "good at this, bad at that." It's more like a coastline, all nooks and crannies. These models are remarkable at some tasks and surprisingly poor at others that look just as hard. They write production code with tools like Claude Code and Codex, then fumble a problem of similar complexity in a different domain.

Why that matters for your roadmap

The frontier also moves. Last November we got a wave of model releases, Claude Opus 4.5 and GPT-5.2 among them, and reasoning over long, multi-step problems took a visible step up. Work somebody had carefully engineered a few months earlier became trivial overnight. You can't lock a three-year roadmap to what the models can't do today. You have to keep testing the specific shape of your problem.

The question to put to your team: where have we actually tested what AI can do for our problem, versus assumed it?

2. Your software stopped being deterministic

Since the 1940s we've built software the deterministic way: same input, same output, write a test, trust the result. That's the ground every engineering team stands on.

LLMs broke it. They're probabilistic. Ask the same question twice and you can get two different answers. That's where hallucinations and the other strange behaviors come from, and it's the source of most "we tried AI and it didn't work" stories. The fix is to constrain the outputs with guardrails and evaluations, and to get the whole team comfortable with non-determinism. It isn't only an engineering skill. Your PMs, your QA people, and your support staff all have to develop the instinct.

3. Where the money should go

When a CEO asks me where to put their AI budget, I reach for a 2x2. I sketched this one live during the session, bad handwriting and all. It's a crude tool for a complex decision, but it's a good place to start.

One axis is business impact: does this move a number the board cares about? The other is execution readiness: do you have the right data, the guardrails to judge the output, and a team that's ready for probabilistic systems?

The top-right corner, high impact and high readiness, is the slam dunk. It's also rare, and if you're sitting in it you don't need us. The quadrant that earns the highest return is the awkward one: high business impact, low execution readiness. I call it preparing the ground. Money spent there moves the project toward the corner where it can actually succeed.

This is why so many pilots demo beautifully and never reach production. The demo isn't the problem. The environment around it isn't ready, and you can't build a house on quicksand.

The question to put to your team: which quadrant are our three biggest AI bets honestly in?

4. Four things AI is actually good at

Across our client work, almost every use case lands in one of four buckets:

• Knowledge management: finding and synthesizing what the company already knows.
• Document processing: vision and language models reading messy, varied documents, well past old-school OCR.
• Conversational interfaces: the chatbots and copilots.
• Workflow automation: stitching the others together, often where agents finally earn their place.

I probably come across as someone who hates chatbots. I don't, but I think we badly over-invest in them. Chat is a hard interface to use well. It hands the burden of learning to the user, who stares at a box and wonders what to type. Most of the public frustration with AI comes from exactly this: trying to rebook a flight with a bot that wastes your time.

What we under-invest in is the unglamorous work: studying the messy human workflows where people actually get stuck, then bringing AI to bear there. It's harder and less tangible than shipping a chatbot. It's also where the real ROI lives.

The question to put to your team: is this a chatbot because the problem needs one, or because it's the easy thing to ship?

5. Why the mandate backfires

Boards set direction, and that's their job. But "become an AI-native company, use AI in everything" is a mandate that measures the wrong thing. Once a metric becomes a target it stops being a useful metric. People chase tokens burned and seats filled, not work that got better.

There's research worth knowing here. BCG's 10-20-70 split says roughly 10 percent of your AI investment should go to algorithms, 20 percent to technology and data, and 70 percent to people and process.

Most portfolio companies invert it. They spend 70 percent on tools and wonder why adoption stalls. The 70 percent is the part nobody photographs for the board deck: preparing your data, training your people, defining what success looks like, and building the measurement to know whether you got there.

A companion to "How to Build & Run AI in Production," the second webinar in our series with Andrei. The full recording is below.

About ten minutes into the webinar, I went to share my screen to show a simple diagram of an agent loop. Zoom refused. I sat there clicking. "Obviously," I said, "my agentic workflow of being able to use Zoom is failing today."

It got a laugh from Andrei, my long-suffering co-host, and it also made my point better than the slide would have. The demo is the easy part. The thing that runs reliably, in front of real users, when the model changes underneath you and the Zoom gods are angry: that's the hard part. And it's the part almost nobody budgets for.

So this is the companion piece to that session. Andrei asked the questions; I sketched the answers. If you watched it, this is the recap to keep. If you didn't, start here.

The check is signed. Now what?

The pattern we see constantly: a company runs an AI pilot, it goes well, everyone is hyped, and then the project stalls for months. There was a much-quoted MIT study last year claiming most AI projects never deliver value. The method was a little dubious, but directionally it's correct.

The reason is simple. A pilot is built to show the art of the possible, all about what the model can do. Production is about everything around the model: making the thing reliable, observable, cost-efficient, safe. I'd put it at around 70 percent of the work, and almost none of that is the AI. It's the scaffolding. If your team is celebrating the demo, treat it as week one. Because it is.

On agents, and why you probably want a workflow

What an agent actually is

Every keynote this year is about agents, so it's worth being precise about what one actually is.

An agent is a loop. Normal software takes an input, does some processing, and returns an output. An agent uses the model to run in a loop:

• it thinks about what needs doing
• it picks an action: call a database, use a tool, check its own memory
• it looks at the result
• it decides whether it's done, and keeps going until it is

The new ingredient is autonomy. The model picks the next step.

Why the last year changed things

A year ago that was a lab demo. Now it's in production at our clients, and a few things changed to make that real:

• Reasoning models. Giving the model the ability to think before it answers got good late last year, with Claude Opus 4.5 and the latest from OpenAI.
• MCP (Model Context Protocol). A standard way to give a model tools, a sort of USB-C for connecting agents to your email or your database.
• Skills. Reusable shortcuts an agent loads when it needs them.
• Tool access. Companies started shipping command-line tools and documentation an agent can actually use.

But you probably still want a workflow

All of which makes agents viable. None of which makes them the default. Most enterprise work already has a well-understood path from input to output, and the honest test is this: can you draw the flowchart?

If you can, build a workflow. A workflow is a set of steps with some branching logic, where each node can still be an AI call doing real work (extraction, classification, scoring), but you decide what happens at each step. You can reason about it, test it, put guardrails in it. Agents do amazing demos. For most production problems you still want the more deterministic, more understandable version. Save the agent for the genuinely unbounded problems, the ones where the path varies every time and you can't draw it in advance.

The question to put to your team: did we build this as an agent because the problem demanded it, or because the demo looked cooler?

Don't put everything on the biggest model

When you do use multiple models, don't run everything on the most expensive one. Most production systems become a hierarchy. A planning model sits at the top, figures out what the user is trying to do, and routes the request. Cheaper models underneath do the actual work. Capability, cost, and latency all drop as you go down the pyramid. This is where people struggle most: they reach for the best model everywhere and watch the bill balloon. Even an agent running on a cheap model can still call tools.

Evals are not optional

Why probabilistic systems need them

None of this matters if you can't tell whether it's working. That is what evals are for: QA for the probabilistic world.

The math is unforgiving. A single model call you can't fully predict. Chain a few together and the variability multiplies: two steps at 80 percent accuracy give you 64 percent overall, not 80. An eval is a labeled dataset aligned to a metric (accuracy, tone, safety, whatever matters) plus a framework that runs it. You run it in three places:

• during development
• before you deploy
• while the system is live, by sampling 20 or 30 percent of real traffic

That last one matters because things change underneath you: an engineer tweaks a prompt, you upgrade a model, the database starts running slower.

The cultural part nobody budgets for

We have teams in India building a mental-health companion for a US company. What reads as appropriate in one context doesn't in the other. So your evals need people with the right context labeling the data, and you often have to reassure your subject-matter experts that tagging this data is about efficiency, not about replacing them. That can be a hard conversation. Evals take more time than writing the code does. Call it 60 to 70 percent of development.

The question to put to your team: show me the golden set (the labeled examples you grade the system against), and tell me when it was last updated.

Product is the new bottleneck

The bottleneck has inverted

For twenty years the bottleneck was writing code. Product managers stayed a few sprints ahead, and engineering caught up. With Claude Code, Cursor, and the rest, that has inverted. Some of our clients tell us their engineers delivered half a year of roadmap by the end of the first quarter. Now product can't write specs fast enough to keep the engine fed.

So the line between product and engineering is blurring. Engineers write specs to hand to their coding agent. PMs stop writing fifty-page PRDs and start handing over working prototypes built in Claude or Lovable. It is worth questioning the whole ritual: if a PRD is written by an AI and read by an AI, why write the PRD? Maybe you build three variations and A/B test them on real users instead of interviewing people about which one they think they'd prefer.

Designing for something that never behaves the same twice

The harder half is the interface. You are designing something that doesn't behave the same way twice. Naive chatbots are high-friction, and people hate talking to them. So the first question I ask is: why not just use a form? If you need flexibility, can you get it from a dynamic but familiar interface instead of an open text box? And if you genuinely need chat, you have to design how the user learns what the system can do, how they recover when it gets something wrong, and how they actually act on a response. Now add low-latency voice, where there are no buttons and no easy undo, and every one of those problems compounds. This is one of the genuinely unsolved problems right now. I don't think anyone has cracked it.

Cost is the thread running through all of it

Software used to be zero-marginal-cost. AI brings marginal costs back, and that is a business-model question, not just an engineering one.

Measure at three levels

System health is latency, error rate, cost per request. Product usage asks whether people are using the thing, and using it the way you intended. Someone who loves chatting with your support bot about their hobbies is a highly engaging product that isn't doing its job. And business impact is the ROI, tied to objectives only you can define.

The trap is token-maxing. "We burned a million tokens today." For what? You could have two agents play tic-tac-toe a million times and burn the same tokens. Goodhart's law: when a measure becomes a target, it stops being a good measure.

The levers, in order

Roughly in the order I reach for them:

• use the right model for each job
• turn on prompt caching
• batch anything that doesn't need to run in real time
• size your context properly

The question underneath all of them is the uncomfortable one: should this be an AI project at all, or is it a few if-else statements in disguise?