Credit: Oxylabs
Frontier AI models have converged within 5% of each other (Stanford 2026 AI Index). Gartner predicts 40%+ of agentic AI projects will be canceled by end of 2027. Oxylabs SVP Gediminas Rickevičius argues the differentiator has shifted from model selection to data infrastructure: web indexes built for agents (structured content, not blue links) and real-time access layers for dynamic information. McKinsey finds 88% of organizations use AI but only 6% are high performers.
In 2026, the way we talk about AI is beginning to change. Two years ago, every boardroom argument circled the same question: which model do we bet on? Today, that question barely registers. Frontier systems have converged so tightly that, according to Stanford’s 2026 AI Index, leading models gained roughly 30 percentage points in a single year on key benchmarks and now cluster within a hair’s breadth of each other on most tasks. The model is no longer the variable; something else is.
That something is data, specifically, what the model sees, when it sees it, and how well it is structured. What makes the difference now is more fundamental: the quality, freshness, and structural depth of the information a model receives. Organizations are running into two walls at once. One is hit by the AI agents, they continue to produce confident errors. The other is long-term and more foundational, finding fresh information requires a new generation of search infrastructure. Both walls lead back to the same foundation: data.
Agentic AI – systems that can plan, search for data, use tools, and execute multi-step tasks with limited human oversight – is being deployed across competitive monitoring, pricing intelligence, market research, procurement, and lead qualification. Sadly, agents often fail at most of these.
Gartner predicts that over 40% of agentic AI projects will be canceled by the end of 2027, citing escalating costs, unclear business value, and inadequate risk controls. A 2025 MIT NANDA study put it even more starkly: roughly 95% of generative AI pilots have failed to deliver measurable returns, with brittle workflows and a lack of contextual learning cited as core reasons.
The instinct, when an agent produces a confident error, is to blame the model – a flaw in reasoning, a hallucination. In reality, the failure can usually be traced to something much simpler: the agent was working with outdated or incomplete information. Sound logic – stale facts, to put it plainly. An agent assessing a competitor’s pricing strategy is limited by whether it can access current and geographically accurate pricing, not by whether it can reason about it. A fleet of such agents across business-critical functions is a structural liability no amount of prompt engineering can fix.
There is a persistent tendency in enterprise AI budgeting to treat data acquisition as overhead, something that lives below the line and, once handed to IT, shouldn’t surface again in strategy conversations. This is no longer tenable. The industry has even coined a term for the discipline of fixing it: context engineering, which, in late 2025, Anthropic defined as the practice of curating the optimal set of information available to a model at inference time.
For organizations whose AI depends on web data, product listings, financial disclosures, news, job postings, regulatory updates, competitor activity, the raw information is out there. The difficulty is in accessing it consistently. Dynamic content rendering, inconsistent data formats, and other common website features become obstacles when you need to extract public data quickly and at scale. Organizations that underinvest in solving them let convenience determine what their AI knows about the world.
To feed an agent fresh information, you need a web index: the organized database search engines consult instead of the live internet. It functions like a library catalog, allowing systems to locate information across billions of pages in milliseconds without reading every “book” from scratch.
The process is continuous: crawlers discover pages, parsers strip away noise like ads and code, and the system stores clean content with quality signals for ranking. While traditional indexes provide links for humans to click, agents require structured data they can use immediately. Standard indexes force agents to fetch, clean, and summarize pages individually, adding delay and token costs. Agent-specific indexes skip this by storing ready-to-use content with verifiable sources, turning search into a reliable foundation for action rather than a list of blue links.
Where it gets interesting for anyone building with AI is that traditional search indexes were designed for people: a title, a link, a short snippet, enough to decide which blue link to click. An agent does not want a list of links. It wants the actual information, already pulled out and structured for a language model.
When an index only hands back links, the agent has to fetch each page, strip out the useful parts, and summarize them itself, which adds delay, drives up token costs, and creates more places to break. Indexes built for agents skip that scramble by storing content the model can use straight away, with source and date attached so every fact can be traced back. That last part matters, because showing where a claim came from is one of the more practical defenses against an agent confidently making something up. So, when someone says agents need better data, it partly means that agents need indexes built for them.
A good index gets you a long way, but even the best one is a snapshot. It reflects the web as of the last crawl, and for a lot of what agents do, that lag is the difference between a right answer and a useless one.
Model commoditization used to be a fringe prediction, but it is not anymore. McKinsey’s 2025 State of AI survey found that 88% of organizations now use AI in at least one function, yet only 6% qualify as high performers extracting meaningful enterprise-wide value. Adoption is no longer the constraint, whereas execution is. And a big part of execution is what the model is fed when it goes to work, which comes down to the infrastructure around it rather than the model itself.
Such infrastructure serves two core purposes: an index to store structured web content for AI models, and live access to reach real-time web data. Standard indexes miss immediate updates like dynamic pricing, regulatory changes, or new product launches, requiring agents to navigate obstacles such as dynamic content and varied formats. Consequently, competitive edge depends less on model selection and more on the breadth and immediacy of the data an AI can access.
Models matter, and building agents is genuinely demanding work. But as models become more capable and more similar, the question of where to direct the next dollar deserves more honest scrutiny than it usually gets. Deloitte’s 2026 State of AI in the Enterprise report says it plainly: companies moving from pilot to production are those investing in the underlying foundations, not just the visible AI layer.
More capable models on their own will not rescue an agentic deployment starved of fresh information. The organizations that capture durable value will be the ones that pair good models with two things working together: indexes built for agents, and live access to the current web for everything those indexes cannot have caught yet. An agent is only as good as the information it can reach at the moment it acts. Closing the distance between what agents can reason about and what they can actually see is what comes next for AI, and building the data infrastructure underneath is how we get there. That shift is already underway, and agentic search is about to change because of it.
Get the most important tech news in your inbox each week.
Contributed article. Not produced by the TNW newsroom and does not reflect the editorial stance of TNW.
