The GPU Shortage Created a New Cloud Economics
The explosion of GPU demand for AI training created a new class of cloud providers whose economics look nothing like traditional cloud computing and more like airlines.
"Even ignoring the lack of a moat regarding GPU clouds, the true driver of this boom in new providers is the total cost of ownership equation for CPU servers versus GPU servers. GPU clouds are dominated purely by capital costs." Dylan Patel, SemiAnalysis
Traditional cloud economics are driven by a complex mix of factors: power usage effectiveness, datacenter design, networking, storage, and compute costs are all significant. Google, Amazon, and Microsoft spend billions optimizing PUE ratios and building custom infrastructure. But GPU cloud economics are radically simpler: Nvidia's extremely high margins mean that 80% of total cost of ownership for a GPU server comes from the capital cost of the GPUs themselves. A relatively poor datacenter operator can buy H100 servers with 13% interest rate debt and still arrive at an all-in cost of $1.53 per GPU hour, while market rates are $2-3+.
This math explains why over a dozen pureplay GPU clouds appeared almost overnight. GPU clouds do not need advanced database services, block storage, multi-tenancy security, or even virtualization. The workloads are homogeneous, the servers are committed for long periods, and storage and networking are tiny costs relative to the GPUs. Capital is the only barrier to entry, not infrastructure sophistication.
But the parallel to airlines is sobering. Aircraft are purchased from one of two manufacturers, capital costs dominate, customers are not locked in, and operators must maximize utilization of expensive assets. Absent monopoly, most airlines barely earn their cost of capital. GPU clouds face an additional risk: GPU generations turn over every two to three years, and nobody will pay 2024 rates for H100s when Blackwell is available. The hyperscalers, with their cash-printing machines and ability to develop custom silicon, hold the long-term structural advantage.
Takeaway: The GPU cloud boom is a gold rush enabled by Nvidia's margins and AI demand but like all commodity businesses, the winners will be those who lock in long-term contracts and low capital costs before the next generation arrives.
See also: Cloud Economics Are Not What They Seem | CUDA Is a Moat Not Just a Library | Custom Silicon Will Eat General Purpose Computing