Meta details Prometheus, the Ohio supercluster heading for 1 million GPUs

Meta’s Prometheus supercluster in New Albany, Ohio, already runs hundreds of thousands of Nvidia GPUs across several buildings, and Meta plans to grow it to about 1 million. The company set out the details in an engineering blog post on Monday.

The first phase of the gigawatt-scale cluster is finished. Meta used it to train its Muse family of models, including the Muse Wonder models. Two years ago, the site was a conventional data centre running web traffic.

“At full scale, Prometheus will comprise roughly 1 million distributed GPUs, all functioning in lockstep,” Meta wrote.

Three generations of Nvidia racks

Prometheus runs two Nvidia platforms today, both built around racks of 72 GPUs that act as one accelerator. Catalina uses the GB200 NVL72, and Clemente uses the newer GB300 NVL72. A third platform, San Miguel, will add Nvidia’s Vera Rubin NVL72 racks.

The GPUs sit in traditional data centres, AI-optimised buildings and rapid-deployment structures, linked by miles of fibre. Each building’s network connects up to 20,736 GPUs. A separate aggregation network joins the buildings over distances from 2km to more than 100km. Meta also uses capacity leased from partners.

From 1,000 interruptions a day

At this scale, a fault on a single machine can stop a whole training run. In mid-2025, during the first phase, interruptions peaked at nearly 1,000 events per 8,000 GPUs each day, Meta said. A year of engineering work has cut that by 99%.

The biggest early problem was signal quality on more than 600,000 NVLink connections between GPUs. Memory faults became a larger issue as Meta scaled up the GB300 racks.

During Muse Wonder’s training, the cluster kept more than 85% of its pre-training time productive, and more than 90% for reinforcement learning. It ran about 47,000 GPU hours on average between interruptions, now above 54,000 after firmware fixes. Meta’s goal is about 100,000.

Reliability also costs hardware. Each 72-GPU rack runs a 64-GPU slice and keeps eight GPUs as spares, and Meta holds about 2% of racks in reserve. A training run on 32,768 GPUs therefore needs 37,656, about 13% more.

Power swings of tens of megawatts

When tens of thousands of GPUs compute and communicate in step, the cluster’s power draw swings by 30% to 40%. Starting or pausing a large job can shift tens of megawatts in seconds. Meta caps GPU power gradually with software it calls Dynamo Dimmer, backed by batteries at rack, facility and grid level.

Meta says it pays the full cost of the energy it uses in New Albany and matches it with new clean and renewable energy. In July, it left the RE100 pact as its gas build-out for AI outpaced renewables.

Its next supercluster, Hyperion in Richland Parish, Louisiana, is being designed with lessons from Prometheus, Meta said. Meta has also pledged $1bn to data centre communities.