What It Really Takes to Train a Frontier LLM

Three years ago, the question "what does it take to train a frontier model?" had a one-word answer: GPUs. That answer is now wrong, or at least incomplete enough to be misleading.

The frontier has moved through three regimes in quick succession. It was chip-bound through 2023, when whoever could get H100s won. It became capital-bound in 2024 and 2025, when whoever could write nine- and ten-figure checks won. And in 2026 it is becoming power-bound: the binding constraint on the next order of magnitude is no longer silicon or money but gigawatts of firm electricity, and the years-long interconnection queues that stand between a signed lease and a running cluster.

A few load-bearing numbers frame everything that follows. All are public estimates, not disclosures:

• A single frontier pretraining run in 2026 is estimated at $200M to $500M for the GPT-5 / Gemini class, on training compute roughly between $10^{26}$ and $10^{27}$ FLOP. Extrapolating the historical trend, Cottier et al. (2024) project that the largest runs will exceed $1B per model by 2027.
• The working unit of frontier compute is now a cluster of 100,000+ accelerators. xAI's Colossus reportedly scaled from 100k to 200k H100-equivalents through 2025; Meta's internal Hopper-class capacity reportedly exceeded 350k H100-equivalents by year-end.
• Training compute is the minority of total cost of ownership. Over a model's deployed life, inference frequently exceeds training spend, and data and talent now rival or surpass raw compute as cost drivers.
• For most organizations, the question is less "should we train from scratch?" and more "where in the stack do we add the most value?" Continued pretraining, post-training, and distillation tend to deliver most of the practical gains at a fraction of the cost of a full pretraining run.

This piece walks the full stack: what "from the ground up" actually means, then the four scarce inputs (compute, power, data, talent), a total-cost model, and a decision framework for who should build what.

Methodology note. Every figure below is drawn from public reporting and third-party analysis: Epoch AI, the Stanford AI Index, the IEA, NVIDIA's own filings, levels.fyi, and industry trade press. Numbers for unreleased models are external estimates, not confirmed disclosures. Nothing here reflects non-public information from any single organization. Where the public record is contested, I say so.

1. What "from the ground up" actually means

"Training a frontier LLM" is not one activity. It is a pipeline of five distinct stages, each with its own cost structure, talent profile, and failure modes. Conflating them is the single most common source of bad budget estimates.

1. Data pipeline. Acquisition, licensing, cleaning, deduplication, filtering, tokenization, and mixing. Unglamorous, and increasingly the real moat (see §5).

2. Pretraining. The single largest compute line item, a model learning next-token prediction over trillions of tokens. This is the part people picture when they imagine "training," and it is where the 100,000-GPU clusters and nine-figure electricity bills live.

3. Mid-training / continued pretraining. Domain adaptation, context-length extension, and capability injection (code, math, multilingual) layered on top of a base model. Far cheaper than full pretraining, and where much practical differentiation now happens.

4. Post-training. Supervised fine-tuning (SFT), preference optimization (RLHF/RLAIF/DPO), and reinforcement learning with verifiable rewards (RLVR) for reasoning. This stage has exploded in importance (much of the 2024 to 2026 capability jump came from post-training, not bigger base models), and it carries heavy human cost, not just compute.

5. Evaluation, safety, and inference optimization. Benchmarking, red-teaming, alignment work, then quantization, distillation, and serving optimization to make the model economical to run.

A useful mental model: pretraining buys you raw intelligence; post-training buys you the product. The two have decoupled enough that a strong post-training pipeline on someone else's base model can beat a weak from-scratch effort at a fraction of the cost. Hold that thought; it is the foundation of the decision framework in §8.

2. Compute: the unit economics of the cluster

The scaling backdrop

Frontier training compute grew roughly 4 to 5x per year from 2018 to 2024, and faster than that for the leading releases. To anchor the FLOP numbers: Epoch AI places GPT-4 near $2\times10^{25}$ FLOP, Claude 3.5 Sonnet near $3\times10^{25}$, and Gemini 1.0 Ultra near $10^{26}$. The 2026 frontier sits in the $10^{26}$ to $10^{27}$ band.

A word on the dollar figures, because they are easy to misquote: they depend entirely on method. On an amortized hardware-plus-energy basis, Cottier et al. (2024) put GPT-4's final training run near $40M and Gemini Ultra near $30M, with costs rising about 2.4x per year. Cloud-rental accounting runs roughly 2 to 3x higher, which is why the same models also get quoted around $78M (GPT-4) and $192M (Gemini Ultra, per the Stanford 2025 AI Index). Same models, different yardsticks. When someone cites a training cost, ask which one they mean.

The accelerator generations

The hardware floor keeps moving. The current and near-term landscape:

Two things matter here. First, performance per GPU is rising fast: a GB200 NVL72 rack is marketed at several times Hopper's training throughput, which is the main reason cost-per-unit-capability keeps falling even as total budgets climb. Second, power per GPU is rising in lockstep (700W, then 1,000W, then 1,400W), which is precisely why the constraint is migrating from the chip to the wall socket (§4).

What a cluster actually costs, and why it isn't just the chips

Quoted GPU prices (an H100 around $25,000 per unit) are a fraction of deployed cost. Industry rule-of-thumb adds $5,000 to $50,000 per GPU for power delivery, cooling, networking, and the building. The interconnect is not optional garnish: frontier pretraining requires every GPU to talk to every other GPU at terabytes per second, which means NVLink within a rack and InfiniBand (or equivalent) across the cluster. The all-in capital cost of a 100,000-H100 facility runs into the multiple billions of dollars before a single token is processed.

The systems-engineering tax nobody budgets for

At 100,000 accelerators, hardware failure is a continuous event, not an exception. A GPU, a cable, or a switch fails somewhere in the cluster on a near-hourly basis at scale. The engineering required to checkpoint frequently, detect failures, and restart without losing days of progress is a first-class discipline. Two metrics define a serious team:

• Model FLOPs Utilization (MFU): the fraction of theoretical peak compute actually converted into useful training. Mediocre teams run 30 to 35%; the best push past 50%. That gap is money: a 20-point MFU difference on a nine-figure run is tens of millions of dollars.
• Goodput: useful training progress net of failures, restarts, and stalls. A cluster that's "up" but spending 20% of wall-clock recovering from faults is quietly burning a fifth of the budget.

This is the part outsiders systematically underestimate. The bottleneck at the frontier is rarely ideas; it is the operational excellence to keep a city-block of silicon productively busy for months without interruption.

3. Hardware availability: the supply chain is the strategy

Buying the chips is harder than affording them. As of mid-2026 the binding hardware constraints sit upstream of NVIDIA itself.

The packaging and memory chokepoint. Blackwell and Rubin both depend on TSMC's CoWoS advanced packaging and on HBM memory stacks, both of which have been capacity-constrained for two years. NVIDIA reportedly secured a large majority of next-cycle CoWoS-L capacity, leaving everyone else to fight over the remainder. Teardowns repeatedly find the limiting factor to be HBM substrate allocation rather than GPU die yield. You cannot solve this with money alone on a short timeline.

Allocation, deposits, and queues. Tier-one OEMs have described their first 12 months of Blackwell allocation as fully committed, with the generation effectively sold out into mid-2026 and millions of Blackwell-class units reportedly in order books. Securing supply now means non-refundable deposits 9 to 12 months ahead of ship date, and even after delivery, expect a multi-month lag to get a rack productively online. Procurement has become a multi-year, reserve-in-writing discipline.

Who you are determines whether you can build at all. Allocation flows to the largest, most creditworthy, longest-committed buyers. A well-funded startup with $500M can still find itself behind hyperscalers in the queue. This is the quiet reason so many "frontier" efforts route through a hyperscaler partnership: the partner's allocation, not just its balance sheet, is the asset.

The alternatives are real but narrower than they look. Google's TPUs (Trillium, Ironwood) and Amazon's Trainium are genuine frontier-scale silicon, but largely captive to their own clouds. AMD's MI300/MI350 line is credible on paper and improving, yet the CUDA software moat (a decade of kernels, libraries, and tribal knowledge) keeps most serious training on NVIDIA. For a from-scratch builder, "just use AMD" usually means absorbing a software-maturity tax that can erase the hardware savings.

4. Power and infrastructure: the gigawatt bottleneck

This is the section that has changed most in the past 18 months, and the one most builders still under-weight.

The constraint has physically moved. Chips can be manufactured at scale on a 12 to 18 month cadence. Power infrastructure cannot. Through 2023 the assumption was that GPUs gated AI; in 2026 the gating factor is available megawatts. The scale of the backlog is the tell: the nationwide interconnection queue peaked near 2,600 GW of pending generation and storage at end-2023, more than double the country's entire installed capacity, and still stood around 2,290 GW at the end of 2024, with large projects routinely waiting four-plus years (Lawrence Berkeley National Lab). New transmission lines can take up to eight years to build.

The numbers are staggering and concentrated. US data-center power load is projected to nearly double from ~82 GW (2025) to ~153 GW (2028) (McKinsey's The cost of compute, as reported by Bloom Energy). Globally, the IEA's Energy and AI report puts data-center electricity near 415 TWh in 2024, rising toward ~945 TWh by 2030, about the annual consumption of Japan. For scale, one gigawatt is roughly the output of a large nuclear plant, increasingly consumed by a single campus.

Density compounds the problem. Rack power density has jumped from 10 to 14 kW to over 100 kW as accelerators have grown, forcing fundamental redesigns of electrical distribution and a wholesale shift to liquid cooling. In less efficient or legacy facilities, cooling alone can consume 30 to 40% of total load; even in optimized, liquid-cooled AI campuses, heat rejection now drives facility design. The facility is no longer a shed for servers; it is a power plant with compute attached.

How the leaders are responding, and what it signals. The hyperscaler playbook has converged on two moves: first, chase power-rich geographies (Texas is projected to pass 40 GW by 2028; new multi-billion-dollar campuses are landing in Louisiana, the Gulf, and the Middle East), and second, bypass the grid with on-site generation, dedicated PPAs, and "energy islands." Regulators are reacting in turn: Texas's SB-6 now imposes new disclosure, study fees, and financial commitments on loads above 75 MW.

The takeaway for a builder: site selection has been demoted from a real-estate decision to the strategic decision. The question is no longer "where is land and fiber?" but "where can I secure a gigawatt of firm power inside my timeline?" If you cannot answer that, the GPU procurement conversation is premature.

5. Data: the wall, the moat, and the synthetic escape hatch

The data wall is real and arriving now. Villalobos et al. at Epoch AI estimate the effective stock of public human-generated text on the order of $3\times10^{14}$ tokens (with a wide confidence interval) and project that frontier runs will consume it somewhere between 2026 and 2032, with a median around 2028, sooner if models are overtrained. The signal is already visible: practitioners increasingly report that data quality and freshness, not raw compute, is the limiting factor on the next increment of capability.

Quality beats quantity, decisively. The clearest public demonstration: Meta trained Llama 3 on 15 trillion tokens, over 7x the 2 trillion behind Llama 2, and reported that even its smallest 8B model kept improving log-linearly well past the point where compute-optimal scaling laws predicted diminishing returns (Meta's Llama 3 release). More, better-filtered data kept paying off. Curation discipline (dedup, filtering, mixing, ordering) is now where a disproportionate share of frontier differentiation is manufactured, and it is the least visible line item in any budget.

Three escape hatches from the wall:

1. Licensing. Frontier labs have signed nine-figure content deals (publishers, forums, media archives) to secure proprietary, legally clean corpora. This is increasingly a legal-and-BD function, not just an engineering one, and an underappreciated cost center.
2. Synthetic data. The Phi line showed that small models trained on curated "textbook-quality" synthetic data can punch far above their weight. But the lesson is discipline, not volume: naively training on AI-generated data degrades quality across generations ("model collapse"). Synthetic data is a scalpel, not a firehose.
3. New modalities and data efficiency. Video, audio, sensor, and interaction data, plus algorithmic gains that extract more capability per token, are the long-run answer once text plateaus.

The human-annotation cost is the buried headline. For post-training, high-quality human (and increasingly expert) labeling, such as preference data, reasoning traces, and domain expertise, has become so expensive that some analyses now put annotation cost above raw compute cost for certain frontier efforts. The "free internet text" era is over; the next era is paid, curated, and partly synthesized.

6. Talent: a market of a few thousand people

The pool is genuinely tiny. Estimates put the number of people worldwide who can credibly operate at the training frontier at roughly 10,000 to 20,000, and the binding constraint is the credential pipeline, which grows far slower than demand. One telling statistic: even amid an all-out comp war, ~70% of senior AI roles reportedly stay open for more than 90 days. More money is not minting more qualified people fast enough.

Compensation has decoupled from every prior norm. Public datapoints (levels.fyi, mid-2026): OpenAI L5 engineers around $1.15M total comp; Anthropic medians around $600K, with senior ICs well above. At the extreme tail, reported packages run into the hundreds of millions, and at least one widely-reported deal approached $1.5B over six years. These are signing-the-scarce-asset numbers, not labor-market numbers: as the framing goes, $10M for an engineer is rounding error against a $1B training run.

What a real frontier team looks like. It is not one kind of person. A serious from-scratch effort needs, at minimum: pretraining researchers, a distributed-systems and infrastructure cohort (often the true bottleneck), data engineers, post-training and RL specialists, evaluation and safety researchers, and the SRE muscle to keep the cluster alive. The systems and data people are frequently harder to hire than the headline "AI researchers," precisely because the market over-indexes on the latter.

The retention reality. As the market matures, the dominant retention conversation at frontier labs is shifting from comp to burnout, mission, and research environment. Several high-profile researchers have walked away from eight-figure packages for better infrastructure or a clearer mission, a reminder that at this tier, money is necessary but not sufficient.

7. Putting it together: a total-cost-of-ownership model

Here is an illustrative stack for a single frontier-class pretraining effort in 2026. Treat the ranges as order-of-magnitude, not a quote; actuals swing widely with scale, efficiency, and whether you own or rent.

Four implications that change the strategy:

1. The headline GPU number is a trap. It is frequently less than half of true cost once failed runs, data, talent, and facilities are included.
2. Inference is the long-run cost center. Over a deployed model's life, serving frequently exceeds the entire training budget, which is why distillation and serving efficiency are first-order, not afterthoughts.
3. Models depreciate brutally. A model trained in 2024 is stale by 2026. The frontier is a treadmill of continuous retraining, not a one-time build: a capex pattern closer to semiconductors than software.
4. Capex vs. opex is a fork in the road. Building gives you control and (eventually) unit economics; renting cloud capacity gives you speed and optionality at a 2 to 4x premium per GPU-hour but no multi-billion-dollar facility bet. The right answer depends entirely on scale and duration of need.

8. The decision framework: who should actually build what

This is the section to internalize if you take nothing else away. "Train a frontier model from scratch" is the right answer for a vanishingly small number of organizations. The honest decision tree:

Tier 1: Train a frontier base model from scratch. Who: A handful of well-capitalized labs and hyperscalers with secured power, allocation, and a frontier team. When it makes sense: You are competing to define the frontier itself, you have $500M+ to commit per generation, and you can absorb the treadmill of continuous retraining. For everyone else, this is almost always the wrong choice.

Tier 2: Continued pretraining / mid-training on an open base. Who: Organizations with a strong domain corpus (legal, biomedical, financial, code) and serious data. Cost: Single-digit to low-tens of millions. Why: You capture most domain value without paying the pretraining tax. Vastly underused relative to its leverage.

Tier 3: Post-training (SFT + preference optimization + RL). Who: Most teams that think they want a "custom model." Cost: Hundreds of thousands to low millions. Why: This is where the product-relevant behavior lives. A disciplined post-training pipeline on a good open base beats a weak from-scratch effort, full stop.

Tier 4: Fine-tuning, adapters (LoRA, QLoRA), and distillation. Cost: Often $500 to $50K, i.e. 1 to 5% of from-scratch. Why: For narrow tasks, this captures the majority of achievable value at a rounding-error cost.

Tier 5: Just use the API. Why: For the large majority of applications, the correct "training strategy" is excellent prompting, retrieval, and orchestration on a frontier API. No GPUs, no power, no team. The cost discipline is in not training.

The uncomfortable truth: the DeepSeek V3 episode is the proof point that cuts both ways. The technical report puts its final training run at 2.788M H800 GPU-hours, about $5.6M at an assumed $2 per GPU-hour rental (DeepSeek-V3 Technical Report). Read that figure carefully: it is the headline run only and explicitly excludes prior research, failed runs, ablations, and staff, so it is emphatically not what it would cost you to reproduce. Even with that caveat, it shows that algorithmic efficiency can collapse the compute bill by orders of magnitude. The lesson is not "training is cheap now"; it is that brute-force scale is one strategy, and increasingly not the most capital-efficient one.

9. Outlook: the power-bound era

Three forces define the next 24 months.

Scale keeps climbing, but slower, and for different reasons. The frontier marches from $10^{26}$ toward $10^{27}$ FLOP, and on current trends the largest runs plausibly cross $1B per model by 2027. But the deceleration in the growth rate is driven by power, capital, and data, not silicon. The chips are, for once, the easy part.

Efficiency is the counter-current. Cost per unit of capability keeps falling sharply via better hardware, better algorithms, and better data curation. Infrastructure costs have reportedly dropped on the order of hundreds-fold over the past several years, and inference costs fall roughly an order of magnitude per year. The absolute frontier gets more expensive while everything below it gets dramatically cheaper, which steadily widens Tiers 2 through 5 above.

The risk register is real. A correction at the top of the talent market looks likely. Aggressive depreciation schedules on rapidly-obsoleting GPUs are a balance-sheet question the industry has not fully reckoned with. Grid strain is producing residential rate increases (8 to 15% in some data-center-heavy communities) and a political backlash that could throttle the buildout. And the data wall, synthetic-data limits, and the open question of whether transformer scaling continues to pay off all sit on the horizon. None of these are disqualifying. All of them belong in a serious plan.

The bottom line

Training a frontier LLM from the ground up in 2026 is a $200M to $500M+ undertaking per model, gated less by money than by firm power and chip-packaging allocation, dependent on a few thousand specialists and an increasingly paid, curated, and synthesized data supply, and shadowed by an inference bill that outlasts and often outweighs the training run itself. The frontier has become a power-and-capital problem wearing a compute costume. For the handful of players defining that frontier, the prize justifies the spend. For virtually everyone else, the winning move is to build on the frontier, through continued pretraining, post-training, and distillation, rather than trying to recreate it. The most valuable skill in the field is no longer "can you train a giant model?" It is "do you know which 5% of the stack actually needs to be custom?"

Sources and notes

Every figure in this piece traces back to public reporting. The ones doing the heavy lifting:

• Cottier et al. (2024), The rising costs of training frontier AI models — the training-cost methodology and the amortized vs. cloud-rental distinction. arxiv.org/abs/2405.21015
• Epoch AI — frontier training-compute and FLOP trends. epoch.ai/trends
• Villalobos et al. (Epoch AI), Will we run out of data? — the data-wall analysis. arxiv.org/abs/2211.04325
• Stanford HAI, 2025 AI Index Report — cloud-rental training-cost figures. hai.stanford.edu
• IEA, Energy and AI — global data-centre electricity demand. iea.org/reports/energy-and-ai
• Lawrence Berkeley National Lab, Queued Up — interconnection-queue capacity. emp.lbl.gov/queues
• McKinsey, The cost of compute — US data-center load projection (~82 GW to ~153 GW by 2028). mckinsey.com (also reported by Bloom Energy).
• Meta, Introducing Llama 3 — the data-scaling result. ai.meta.com/blog/meta-llama-3
• DeepSeek, DeepSeek-V3 Technical Report — the efficiency cost figures. arxiv.org/abs/2412.19437
• NVIDIA — investor relations and quarterly filings, for allocation and shipment commentary. investor.nvidia.com
• levels.fyi — compensation data for AI and ML roles. levels.fyi

A note on the numbers: the cost, comp, and unreleased-model figures are external estimates and reported deals, not confirmed disclosures, and methods differ enough that two honest sources can quote costs 2 to 3x apart (see §2). Nothing here draws on non-public information from any company; it is a reading of the public record.