This is the eleventh part in a series on how AI is crossing out of software and into the physical world. This time, Jan Bosch looks at when software stops answering and starts acting. The promise is easy to state. Software that doesn’t merely answer questions but executes workflows: reading the ticket, checking the inventory, issuing the credit, updating the record, sending the mail. Not a tool a person uses, but a colleague-shaped process that runs. Gartner’s assessment of how that’s going is bracing. The firm predicts that more than 40 percent of agentic-AI projects will be cancelled by the end of 2027, citing escalating costs, unclear business value and inadequate risk controls. It also estimates that of the thousands of vendors now marketing agentic AI, only around 130 are actually doing it – the rest being chatbots and RPA with a new label, a practice Gartner calls agent washing. I could write a whole piece from that paragraph, and it would be the wrong piece. The cancellations are a symptom. What I want to spend this one on is the arithmetic underneath them, because once you see it, almost everything about agent deployment stops being mysterious. Here’s the arithmetic. Reliability doesn’t add; it multiplies. An agent step that succeeds 95 percent of the time is, in isolation, impressive. Chain five of them and you’re at 77 percent. Chain twenty and you’re at 36 percent. Chain thirty, which is a perfectly ordinary number of steps for a real business process with lookups, validations, approvals and writes, and you’re at 21 percent. Push the per-step reliability to 99 percent and thirty steps still only gets you to 74 percent: a process that fails one time in four. To reach 97 percent on a thirty-step task you need 99.9 percent per step, which is a figure almost nothing in software achieves against unstructured inputs. This is why demos are so misleading, and the mechanism is worth naming precisely. A demo is three steps. Real work is thirty. The gap between them isn’t polish, effort or model maturity; it’s exponentiation. The benchmark data says exactly this, once you look at the right metric. Sierra’s τ-bench was built to test agents on multi-turn tasks with real tool use and written policies, and its most useful contribution isn’t a leaderboard but a measure: pass^k , the probability that an agent succeeds on the same task across k repeated attempts. The original results had GPT-4-class agents succeeding on fewer than half of tasks on a single attempt and on roughly a quarter when the same task was run eight times. Frontier models have since pushed past 80 percent on pass^1 in the retail domain, which is real progress. But pass^1 is the metric that flatters, and the pass^k curves are, in Sierra’s own words, far from ideal. An agent that works four times out of five is a demo. An agent that works four times out of five every time you run it is a product. Enterprises buy the second thing and are shown the first. Which brings me to the design consequence, and to the idea I’ve been circling since the summer. You can’t test your way to confidence in a thirty-step agent by sampling end-to-end outcomes. The state space is too large, the failures aren’t independent and a passing run tells you almost nothing about the next one. So, the architecture that actually survives contact with production isn’t a long autonomous chain; it’s a short sequence of narrow steps, each with a stated contract: what must be true going in, what must be true coming out. And each input and output is checked. This is the fullest form of the argument I’ve been making throughout this series, and in agent systems, it stops being a philosophical position and becomes the only testable unit available. The durable artifact isn’t the prompt, the model or the orchestration graph; it’s the contract at each boundary, plus the running evidence that it held. You can’t verify the whole; you can verify the joints. There’s a second design axis, and it’s the one I see n
Machines that think: agents in the enterprise
汽车电子
英文
This is the eleventh part in a series on how AI is crossing out of software and into the physical world. This time, Jan Bosch looks at when software stops answering and starts acting.
本站仅提供摘要,完整内容请阅读原文(版权归原媒体所有)
https://bits-chips.com/article/machines-that-think-agents-in-the-enterprise/
相关阅读
11-02 19:04
11-02 16:01
11-02 03:00
11-01 22:00
11-01 20:54