AWS says almost 90% of Amazon’s early AI agent prototypes never shipped

Almost 90% of the AI agent prototypes Amazon teams built two years ago never reached production, AWS vice president of agentic AI Swami Sivasubramanian said. Amazon was then starting to invest heavily in agents.

Sivasubramanian spoke on Wednesday at HumanX in Amsterdam. Citing analysts, he said 17% of organizations have successfully deployed AI agents and only 7% can measure the return on that investment. Customers’ CTOs and engineering leaders tell him their boards ask about ROI, he said.

“I have 15 prototypes for very similar projects. I can’t tell which one to back up,” Sivasubramanian said, relaying what customers tell him.

Five reasons agents stall

AWS sends forward deployed engineers to customer sites to build solutions with them. Over six months, that team traced failed projects to five root causes, Sivasubramanian said. Teams work on the wrong problem, and they cannot measure whether it works. Governance is an afterthought. Leaders freeze between fear of missing out and fear of committing. Nobody redesigns the organization for AI.

OpenAI’s Colin Jarvis told the same conference that enterprise AI is stuck on deployment, not models.

Without a defined business outcome, teams keep improving a proof of concept and never exit, Sivasubramanian said. He compared it to circling a Paris roundabout four or five times on his first drive in Europe. In AI, the fuel is tokens and budget.

From one engineer’s side project to 39,000 users

Amazon’s answer was to let a thousand flowers bloom and measure which ones worked, he said. More than 100,000 Amazon engineers use Kiro, Amazon’s agentic coding tool, as do millions of developers outside the company.

Bolin Chen, an engineer in the personalization team for Amazon stores, built an always-on assistant called MeshClaw on Kiro on nights and weekends. It ran through Slack. After he shared it on an internal Slack channel, it grew from one person to four founders to thousands of contributors across Amazon, Sivasubramanian said.

Amazon’s measurement systems flagged the project. The Kiro team had separately been prototyping always-on agents, persistent memory and multi-agent coordination, so the two groups joined. The result is the open-source project Kiro Crew. Close to 39,000 Amazon employees built on it within 30 days of its internal launch, and it has 500 external contributors, he said.

Kiro also improves itself, he said. A system inside Kiro found that the agent was rereading files that had not changed since the previous run. The agent diagnosed the problem, wrote the fix and validated it, cutting that waste by 83%.

“No human diagnosed the problem. The system itself found it,” Sivasubramanian said.

One golden path, and an agent in a box

Amazon teams had built agents on EC2, EKS, SageMaker and other services, which strained security and infrastructure teams, he said. Amazon consolidated on Bedrock for AI inference and AgentCore for agent hosting, a single path approved by security.

AWS’s open-source agent framework, Strands, now has a feature that puts an agent in a box, he said. The box is a deterministic layer outside the agent that governs which tool calls it can make. As the agent proves itself, the box can widen. Google DeepMind’s Kareem Ayoub told HumanX that companies can fence AI in.

“It’s not probabilistic. It’s always deterministic and mathematically provable that the agent cannot exceed its boundaries,” Sivasubramanian said.

Six developers, 76 days

High-ambiguity projects should start with very small teams, Sivasubramanian said. Nearly 80% of the Fortune 100 use Bedrock, he said, and he called it the fastest-growing AWS service. When demand exploded, six developers re-engineered it in 76 days. He said the work would have taken 30 developers 18 months. The team later grew to 35 developers as the system scaled.

Leaders also need to stay close to the ground, he said. A scientist built the Amazon Quick desktop app overnight with Kiro and showed it to executives within a week. AWS launched it externally within three months. Hundreds of thousands of Amazon employees use Quick, he said.

He told the audience to ask two questions of every AI project. What outcome is it connected to, and is it compounding at every layer?

“Knowing was never the problem. The question is whether you have the rigor and the discipline to go execute it,” Sivasubramanian said.