The hidden costs of renting GPUs you don't own.

Your AI infrastructure bill shouldn't be a mystery your finance team has to solve every month. One AWS cost consultancy found AI infrastructure bills landing close to 3.7x over what teams had originally budgeted, on average. But here's the secret: the bill isn't lying about compute. It's broken into four separate charges wearing one name.

I sat in on a budget review a few weeks back, the kind where someone shares their screen and the room goes quiet for a second. The line item was AI infrastructure spend, and it had grown the same way three months running — up about 20% each month, almost to the decimal. The VP of Finance running the meeting wasn't upset about the size of the number. She was upset that nobody on the team could tell her why it kept moving that way. Usage hadn't grown 20% month over month. The bill had.

The four charges wearing one name

Compute (the only line that's actually compute) — Run the same H100 on a hyperscaler and on a GPU-first cloud, and you're paying wildly different rates for identical silicon. AWS's p5.48xlarge runs about $55 an hour. The same H100 on Lambda or RunPod lists between $2.50 and $3.30 an hour. That's a 2x to 5x spread for hardware that came off the same TSMC line, and a chunk of it has nothing to do with the GPU at all — it's the instance shape forcing you to rent capacity you didn't ask for.

Egress (the toll booth on your own output) — Every major cloud charges nothing to bring data in and real money to let it leave. AWS, Azure, and GCP all sit in the $0.085 to $0.12 per GB range for internet egress. A team running a RAG pipeline can easily push 10 TB a month — call it $900 at the headline rate, closer to $1,300 once NAT Gateway processing is in the mix. None of that shows up on the GPU line.

The idle premium — Cast AI pulled real production telemetry off tens of thousands of Kubernetes clusters and found average GPU utilization sitting at 5%. That's roughly 20 times the capacity most fleets actually need at any given moment, fully billed the whole time. You're billed for capacity provisioned, not work performed.

The burst premium — Most production inference isn't actually unpredictable. But it's still billed at the same on-demand rate as a workload nobody could forecast at all. You're paying full price for an insurance product — burst headroom, multi-region failover, an SLA you've never had to invoke — stacked on top of a hardware markup that has nothing to do with the silicon underneath it.

What that 2x to 5x spread actually is

The hyperscaler-versus-neocloud gap isn't a hardware cost difference — it's the same H100 either way. What you're actually paying for is the abstraction layer wrapped around it: global region footprint, integrated identity and billing, compliance tooling, and the standing option to burst on a moment's notice. That's genuinely worth something to a team prototyping with no ops staff. But for a team with steady, forecastable load, the math gets uncomfortable. One example: a company scaling from $2M to $8M in ARR was paying $23,200 a month in SageMaker infrastructure. Same workloads, right-sized and with abandoned EBS volumes cleaned up, dropped that to $9,400 a month — $164,400 a year that had been buying exactly nothing.

The bill isn't lying about compute. It's four charges wearing one name.

The four charges in this post don't show up as four charges. They show up as one number that goes up faster than your usage does, and most teams never get past "AI is expensive" as the explanation.

Adapted from Mark Tuck's essay, "The Hidden Cost of Renting GPUs You Don't Own" (LinkedIn). Mark Tuck is Private AI Cloud | Strategy & Architecture @ Cloud Ingenuity. Figures cited from public AWS, Lambda, and RunPod pricing and Cast AI production telemetry.