Executive Summary
Training makes headlines; inference makes money. Inference's share of AI compute rose from one-third in 2023 to half in 2025 and two-thirds in 2026, and Bloomberg Intelligence projects a $1.3 trillion inference market by 2032 — double the training market. Training is one-time capex; inference is recurring opex — that is, recurring revenue. This report maps the market's tiers, examines the dedicated-chip challengers, and explains how inference cost now defines AI business models.
1. Why Inference, Why Now: Three Demand Explosions
First, reasoning models devour tokens — models that deliberate before answering consume multiples of the tokens per query. Second, agents: a single task triggers dozens of model calls and tool uses, multiplying inference demand. Third, open-weight proliferation: the freer the models, the greater the demand for compute to run them — as open weights improve, what grows is not the model market but the compute market.
Forecasts vary but point the same way: inference infrastructure from ~$106B (2025) to $255B (2030, 19.2% CAGR), and $1.3T by 2032 (BI). Some estimates put inference at 10–50× training.
2. Market Tiers: Who Takes Margin Where
| Tier | Key players | Competitive axis |
|---|---|---|
| General GPU | NVIDIA (dominant), AMD | Training/inference flexibility, CUDA ecosystem, supply chain control |
| Dedicated inference chips | Cerebras (wafer-scale), Groq (LPU) | Speed and unit cost — Cerebras 3–8× throughput; Groq cheaper per Mtoken (model-dependent) |
| Hyperscaler silicon | AWS (Inferentia/Trainium), Google (TPU) | Price/supply control within own cloud; NVIDIA dependency hedge |
| Inference clouds/APIs | Hyperscalers; Together, Fireworks, etc. | Open-weight hosting, utilization maximization, serving optimization |
| Serving software | vLLM, TensorRT-LLM, etc. | The software layer that decides cost-per-token on identical hardware |
Sources: company disclosures, Spheron benchmarks (Aug 2026), Forbes. Performance and cost comparisons vary heavily by model and workload.
The Cerebras IPO — Emblem of the Inference Bet
One of 2026's biggest IPOs, Cerebras priced at $22–25B and has traded up to roughly $50B — over 50× its $885M 2026 revenue guidance. The premium prices a scenario where dedicated inference architectures erode NVIDIA's general-GPU margins. There is technical substance: on memory-bound inference, the wafer-scale chip hits ~2,500 tokens/second/user on Llama 4 Maverick versus ~1,000 for an NVIDIA DGX B200. The open questions are customer concentration and margin durability — and the right frame is not "Cerebras vs NVIDIA" but capital allocation between training and inference.
3. Token Economics: Inference Cost Defines the Business Model
Total cost per task, not unit price
Half the token price means nothing if the model uses twice the tokens deliberating. Model selection has shifted from price-per-million-tokens to cost-per-completed-task.
Utilization is margin
GPUs burn money while idle. The unit-cost gap between high-utilization providers and enterprises bearing fixed self-hosting costs alone creates the business space for inference specialists.
Software closes hardware gaps
KV-cache compression, batching, and speculative decoding improve cost-per-token severalfold on identical hardware. Beneath the chip war, the serving-software layer is the real battleground.
Strategic Implications
For adopters — inference cost is now the COGS of AI services. Model tiering (cheap models handle volume, frontier models handle escalations), caching strategy, and API-vs-self-hosting break-even analysis belong in financial planning.
For the industry — unlike training, inference is a distributed market sensitive to latency, geography, and data sovereignty, which resists a single winner. NVIDIA's dominance holds for now, but margin pressure advances on three fronts simultaneously: dedicated chips (speed), hyperscaler silicon (cost control), and serving software (efficiency). Falling inference prices are not a crisis — like falling electricity prices, they are the condition for demand explosion.
Need a talk, article, or advisory on this topic?
I deliver executive keynotes, contribute industry reports, and advise on AI strategy.
Get in Touch