The hidden operating cost of AI infrastructure, and one engineer working on it

The AI infrastructure earnings calls of the past eight quarters have given the public a precise vocabulary for what the build-out costs in capital. Hyperscaler GPU procurement. Power purchase agreements. Real-estate footprints. The vocabulary they have not given the public is for what it costs to keep the clusters healthy on a recurring basis after the capital is spent. That line item, on close inspection, has become one of the largest hidden cost centers in the entire build-out. It is growing faster than the capital line above it.

The visible numbers in the AI infrastructure conversation describe the capital story. Hyperscaler GPU procurement is on track to cross multi-trillion-dollar cumulative spend over the current cycle. Power purchase agreements have moved into the range that historically described heavy industry. Real-estate commitments have followed. The capital narrative has been told in detail across two years of investor updates.

The operational story is less visible. It describes what it costs to keep the clusters healthy. The work is unglamorous and largely manual. GPU node failures have to be detected, triaged, and remediated. Pods have to be rescheduled around degraded hardware. Resource utilization across an accelerator fleet has to be monitored, balanced, and reported on. Each of these tasks is, in current production environments, performed by a class of engineer whose compensation is among the highest in the industry.

The scale of the bill is enormous. Operators have long reported that a large share of production GPU capacity sits idle. The headcount required to keep cluster operations running has scaled with cluster size, in proportion rather than sub-proportion, in environments where the explicit goal of every infrastructure team is to break that proportionality. The operational layer, on aggregate, is one of the line items that turns the AI infrastructure thesis from a strong investment story into a structural margin problem.

The work to address it has, until recently, sat inside the bespoke automation tooling of the largest operators, accessible only to the engineers who built it. That is starting to change. Shashidhar Bhat, a software engineer in the big-data infrastructure organization at ByteDance, has spent the past two years producing a body of work that maps directly onto the operational layer the rest of the industry has been describing as a problem.

The pieces, individually, look like ordinary infrastructure components. Custom device plugins for finer-grained accelerator scheduling. Observability tooling built on top of NVIDIA’s Data Center GPU Manager. Autonomous pod rescheduling logic that reacts to hardware degradation without human escalation. Each is the kind of thing that gets shipped quietly inside an internal infrastructure team. Taken together, they describe the operational layer that the industry has been outsourcing to site reliability engineers, ported into software and hardened against production load.

The scale at which Bhat’s work runs is part of what makes it credible as a reference architecture. ByteDance, parent of TikTok, runs Kubernetes at very large scale. The clusters Bhat works on span hundreds of GPU nodes and process roughly one petabyte of data a month. Bhat says an internal agent-based automation system he built has cut GPU idle time by about 35 percent in the environment he works on.

A thirty-five percent figure is, by the operational standards of the field, large. Hyperscaler-class operators have for years been chasing single-digit-percentage improvements in idle rates, on the reasoning that single-digit improvements at hyperscaler volumes pay back in eight figures. A reduction at the scale Bhat reports is the kind of result that, when it appears in production at a peer company, is closely held. The fact that it has been reported at all is part of why the wider operator community has begun paying attention.

The other half of Bhat’s recent work has appeared on the open-source side. He has been a contributor to Kubewharf Katalyst, the open-source resource management framework ByteDance publishes under its KubeWharf project. The Katalyst project is one of the few in the cloud-native ecosystem to address the joint scheduling of CPU and GPU resources under load. The convergence between an engineer’s internal production work and external open-source contributions is the rare kind of pattern the maintainer community recognizes as substantive rather than promotional.

The third leg of the body of work is Carbon-Kube, the open-source Kubernetes scheduler Bhat released this past December with Sathwik Rao Sirikonda, also at ByteDance. The scheduler is a distinct project from his internal ByteDance work and addresses the carbon-emissions dimension of cluster operations rather than the headcount dimension. The project ships with a citation file, a published benchmark methodology, and reproducible scripts. The contribution is methodologically rigorous in a way that most internal infrastructure tooling never bothers to be.

The combined picture is what makes the case worth making at the industry level. The work to address it has been happening quietly inside the largest companies, accessible only to their internal teams. That is changing, in part because of the work of operators like Bhat, whose contributions span internal production deployments, external open-source maintenance, and research-grade publications under his own name.

The argument that the operational layer is the next major margin frontier in AI infrastructure is, on the strength of the work that has shipped in the past year, hard to dismiss. Cluster operators in the next two to three years will need to decide whether to build their own answer or to adopt one of the open-source ones now becoming available. The composition of that answer will reshape the operational margin of every team running production AI workloads.