Meta has published an unusually detailed account of Prometheus, its supercluster in New Albany, Ohio, and of what it costs to keep a frontier training run alive. More than 11 percent of the cluster’s GPUs are either dead or sitting idle as spares waiting to replace the next failure, the company disclosed.
Prometheus is among Meta’s largest compute deployments. It runs Nvidia Grace Blackwell systems today, GB200 and GB300 NVL72 racks, with Vera Rubin racks to follow, and Meta says it will eventually scale to a gigawatt of capacity spanning about one million accelerators across buildings linked over distances from 2 kilometres to more than 100 kilometres. The completed first phase trained Meta’s Muse models, including Muse Wonder.
The failure arithmetic is the point. When Prometheus began operating in mid 2025, Meta was seeing roughly 1,000 interruptions per day for every 8,000 GPUs, where an interruption means something crashed, failed or timed out and forced a rollback to the last checkpoint. Meta says it has cut that rate by 99 percent, to about 11 per day per 8,000 GPUs. Part of the method is redundancy: running a 64 GPU slice in each 72 GPU rack and holding the rest as in-rack spares, plus reserve racks, means provisioning about 37,656 GPUs to sustain a 32,768 active GPU job, roughly 13 percent above the active footprint.
During Muse Wonder training, Meta reports more than 85 percent goodput in pre-training and more than 90 percent in reinforcement learning, with mean time between interruptions of about 47,000 GPU hours, since raised past 54,000. It also described power swings of 30 to 40 percent as tens of thousands of GPUs pause and resume in unison, managed with software that lowers GPU power limits near the site’s electrical ceiling. All figures are Meta’s own, published in its engineering account.