This throughput assumes 100% utilizations. A bunch of things raise the cost at scale:
- There are no on-demand GPUs at this scale. You have to rent them for multi-year contracts. So you have to lock in some number of GPUs for your maximum throughput (or some sufficiently high percentile), not your average throughput. Your peak throughput at west coast business hours is probably 2-3x higher than the throughput at tail hours (east coast morning, west coast evenings)
- GPUs are often regionally locked due to data processing issues + latency issues. Thus, it's difficult to utilize these GPUs overnight because Asia doesn't want their data sent to the US and the US doesn't want their data sent to Asia.
These two factors mean that GPU utilization comes in at 10-20%. Now, if you're a massive company that spends a lot of money on training new models, you could conceivably slot in RL inference or model training to happen in these off-peak hours, maximizing utilization.
But for those companies purely specializing in inference, I would _not_ assume that these 90% margins are real. I would guess that even when it seems "10x cheaper", you're only seeing margins of 50%.
parhamn 2 hours ago [-]
Do we know how big the "batch processing" market is? I know the major providers offer 50%+ off for off-peak processing.
I assumed it was to slightly correct this problem and on the surface it seems like it'd be useful for big data places where process-eventually is enough, e.g. it could be a relatively big market. Is it?
empiko 2 hours ago [-]
You also need to consider that the field is moving really fast and you cannot really rely on being able to have the same margins in a year or two.
lbhdc 2 hours ago [-]
If you are willing to spread your workload out over a few regions getting that many GPUs on demand can be doable. You can use something like compute classes on gcp to fallback to different machine types if you do hit stockouts. That doesn't make you impervious from stock outs, but makes it a lot more resilient.
You can also use duty cycle metrics to scale down your gpu workloads to get rid of some of the slack.
jerrygenser 3 hours ago [-]
Re the overnight that's why some providers are offering there are batch tier jobs that are 50% off which return over up to 12 or 24 hours for non-interactive use cases.
derefr 2 hours ago [-]
> There are no on-demand GPUs at this scale.
> These two factors mean that GPU utilization comes in at 10-20%.
Why don't these two factors cancel out? Why wouldn't a company building a private GPU cluster for their own use, also sit a workload scheduler (e.g. Slurm) in front of it, enable credit accounting + usage-based-billing on it, and then let validated customer partners of theirs push batch jobs to their cluster — where each such job will receive huge spot resource allocations in what would otherwise be the cluster's low-duty point, to run to completion as quickly as possible?
Just a few such companies (and universities) deciding to rent their excess inference capacity out to local SMEs, would mean that there would then be "on-demand GPUs at this scale." (You'd have to go through a few meetings to get access to it, but no more than is required to e.g. get a mortgage on a house. Certainly nothing as bad as getting VC investment.)
This has always been precisely how the commercial market for HPC compute works: the validated customers of an HPC cluster sending off their flights of independent "wide but short" jobs, that get resource-packed + fair-scheduled between other clients' jobs into a 2D (nodes, time) matrix, with everything getting executed overnight, just a few wide jobs at a time.
So why don't we see a similar commercial "GPU HPC" market?
I can only assume that the companies building such clusters are either:
- investor-funded, and therefore not concerned with dedicating effort to invent ways to minimize the TCO of their GPUs, when they could instead put all their engineering+operational labor into grabbing market share
- bigcorps so big that they have contracts with one big overriding "customer" that can suck up 100% of their spare GPU-hours: their state's military / intelligence apparatus
...or, if not, then it must turn out that these clusters are being 100% utilized by their owners themselves — however unlikely that may seem.
Because if none of these statements are true, then there's just a proverbial $20 bill sitting on the ground here. (And the best kind of $20 bill, too, from a company's perspective: rent extraction.)
thenewwazoo 1 hours ago [-]
> Why wouldn't a company ... let validated customer partners of theirs push batch jobs
A company standing up this infrastructure is presumably not in the business of selling time-shares of infrastructure, they're busy doing AI B2B pet food marketing or whatever. In order to make that sale, someone has to connect their underutilized assets with interested customers, which is outside of their core competency. Who's going to do that?
There's obviously an opportunity here for another company to be a market maker, but that's hard, and is its own speciality.
mistrial9 41 seconds ago [-]
Snowflake ?
loocorez 13 minutes ago [-]
Sounds like prime intellect
caminanteblanco 6 hours ago [-]
There was some tangentially related discussion in this post: https://news.ycombinator.com/item?id=45050415, but this cost analysis answers so many questions, and gives me a better idea of how huge the margin on inference a lot of these providers could be taking. Plus I'm sure that Google or OpenAI can get more favorable data center rates than the average Joe Scmoe.
A node of 8 H100s will run you $31.40/hr on AWS, so for all 96 you're looking at $376.80/hr. With 188 million input tokens/hr and 80 million output tokens/hr, that comes out to around $2/million input tokens, and $4.70/million output tokens.
This is actually a lot more than Deepseek r1's rates of $0.10-$0.60/million input and $2/million output, but I'm sure major providers are not paying AWS p5 on-demand pricing.
Edit: those figures were per node, so the actual input and output prices would be divided by 12.$0.17/million input tokens, and $0.39/million output
zipy124 5 hours ago [-]
AWS is absolutely not cheap, and never has been. You want to look for the hetzner of the GPU world like runpod.io where they are $2 an hour, so $16/hr for 8, that's already half of aws. You can also get a volume discount if you're looking for 96 almost certainly.
An H100 costs about $32k, amortized over 3-5 years gives $1.21 to $0.7 per hour, so adding in electricity costs and cpu/ram etc... runpod.io is running much closer to the actual cost compared to AWS.
caminanteblanco 5 hours ago [-]
Ok, so the authors apparently used atlas cloud hosting, which charges $1.80 per h100/hr, which would change the overall cost to around $0.08/ million input and $0.18/million output, which seems much more in line with massive inference margins for major providers.
bluedino 2 hours ago [-]
> A node of 8 H100s will run you $31.40/hr on AWS, so for all 96 you're looking at $376.80/hr
And what stinks is that you can't even build a Dell/HPE server like this online. You have to 'request a quote' for an 'AI Server'
Going through SuperMicro, you're looking at about $60k for the server, plus 8 GPU's at $25,000 each, so you're close to $300,000 for an 8 GPU node.
Now, that doesn't include networking, storage, racks, electricity, cooling, someone to set that all up for you, $1,000 DAC cables, NVIDIA middleware, downtime as the H100's are the flakiest pieces of junk ever and will need to be replaced every so often...
Setting up a 96 H100 cluster (12 of those puppies) in this case is probably going to cost you $4-5 million. But it should cost less than AWS after a year and a half.
188M input / 80M output tokens per hour was per node I thought?
Reversing out these numbers tells us that they're paying about $2/H100/Hour (or $16/hour for a 8xH100 node).
Disclaimer (one of my sites) https://www.serversearcher.com/servers/gpu - says that a one month commit on a 8XH100 node goes for $12.91/hour. The "I'm buying the servers and putting them in COLO rate" usually works out at around $10/Hour, so there's scope here to reduce the cost by ~30% just by doing better/more committed purchasing.
caminanteblanco 5 hours ago [-]
You were definitely right, I updated the original comment. Thanks for your correction!
paxys 5 hours ago [-]
According to the post their costs were $0.20/1M output tokens (on cloud GPUs), so your numbers are off somewhere.
"By deploying this implementation locally, it translates to a cost of $0.20/1M output tokens"
Is that just the cost of electricity, or does it include the cost of the GPUs spread out over their predicted lifetime?
zipy124 5 hours ago [-]
This is all costs included. Thats 22k tokens per second per node, so per 8 h100's. With 12 nodes they get 264k tokens per second, or 950 million an hour. This get's you to roughly $0.2021 per million at $2 an hour for an h100, which is what they go for on services such as runpod.io . (cheaper if not paying spot-price + volume discounts).
dragonslayer56 6 hours ago [-]
” Our implementation, shown in the figure above, runs on 12 nodes in the Atlas Cloud, each equipped with 8 H100 GPUs.”
Maybe the cost of renting?
34679 6 hours ago [-]
I'm confused because I wouldn't consider a cloud implementation to be local.
randomjoe2 5 hours ago [-]
Local doesn't refer to "on metal" anymore to many people
mwcz 5 hours ago [-]
"On metal" is muddied too. I've heard people refer to web apps running in an OCI container as being "bare metal" deployment, as opposed to AWS or whatever hosting platform.
That's silly, but the idea that "local" is not the opposite of remote is even sillier.
ffsm8 5 hours ago [-]
You can run an OCI container on bare metal though. It doesn't stop being run on bare metal just because you're running in kernel namespaces, aka docker container
Lots of people were advocating for running their k8s on bare metal servers to maximize the performance of their containers
Now wherever that's applied to your conversation... I've no clue, too little context ( ｡ ŏ ﹏ ŏ )
okasaki 5 hours ago [-]
In my opinion, if you're running k8s on bare metal, that's "k8s on bare metal" but still "<your app> on kubernetes", not "<your app> on bare metal".
ffsm8 3 hours ago [-]
Sorry, but then your opinion is just plain wrong
Bare metal in the context of running software is a technical term with a clear meaning that hasn't become contested like "AI" or "Crypto" - and that meaning is that the software is running directly on the hardware.
As k8s isn't virtualization, processes spawned by its orchestrator are still running on bare metal. It's the whole reason why containers are more efficient compared to virtual machines
mystifyingpoi 2 hours ago [-]
I think both of you are correct.
Of course, a process running inside Kubernetes Pod, on a baremetal node will show up in `top` if I run it on the node directly. In such terms, it is running directly on hardware.
But when I deploy this Pod, I'm not interacting with the OS in any way. I'm interacting with Kubernetes apiserver, telling it what to run, not really caring about the operating system underneath. In such terms, the application is running "in k8s".
bee_rider 3 hours ago [-]
Bare metal as in, no operating system? Does Linux really get in the way of these LLM inference engines?
ffsm8 3 hours ago [-]
No, as I said in my previous comment: bare metal as in not a virtual machine
Note that this is a term whose meaning has been expanded to refer to non-VPS servers very recently. Bare-metal has traditionally meant "without an operating system." It did not mean "a server that is an actual server," because that was the default.
It also does not always "clearly" have this new meaning. Somebody who is used to running programs directly (with no intermediate OS) on hardware might not understand what you're saying, or might ask you to clarify, and you probably shouldn't feel put upon by a totally understandable misinterpretation.
edit: Especially when you keep repeating "directly on hardware" when you mean "not on a VM." VMs also run on hardware. You're saying that you're only running on one OS instead an OS in your OS.
dtech 5 hours ago [-]
If you do bare metal as not being under a VM it fits. OCI on linux is cgroup so that counts as not a VM I'd say. Or at least it's a layer closer to the metal than a typical VM running OCI images.
I a Java app running on Linux bare metal?
bee_rider 3 hours ago [-]
Local doesn’t need to be “on metal,” but I’m still confused as to what they are saying. Are they running some local cloud system?
monsieurbanana 5 hours ago [-]
I missed that train
vFunct 5 hours ago [-]
My basement server really confused by all this...
DSingularity 5 hours ago [-]
I guess local for him is independent/private.
6 hours ago [-]
ollybee 5 hours ago [-]
H100's can be $2 and hour, so $192 an hour for the full cluster. They report 22k tokens per second, so ~ 80 million an hour, thats $16 an hour at $0.2 per million. Maybe a bit more for input tokens, but it seems a long way off.
zipy124 5 hours ago [-]
I think you mis-read. Thats 22k tokens per second per node, so per 8 h100's. With 12 nodes they get 264k tokens per second, or 950 million an hour. This get's you to roughly $0.2021 per million at $2 an hour.
cootsnuck 2 hours ago [-]
Super helpful to see actual examples of what it (roughly) can look like to deploy production inference workloads, and also the latest optimization efforts.
I consult in this space and clients still don't fully understand how complex it can get to just "run your own LLM".
s46dxc5r7tv8 5 hours ago [-]
Separation of the prefill and decoding layers with sglang is quite nifty! Normally 8xH100 would barely be able to hold the 4bit quantization of the model without even considering the KV cache. One prefill node for 3 decode nodes is also fascinating, nice writeup.
Just in case you have $3-4M lying around somewhere for some high quality inference. :)
SGLang quotes a 2.5-3.4x speedup as compared to the H100s. They also note that more optimizations are coming, but they haven't yet published a part 2 on the blog post.
abdellah123 6 hours ago [-]
Wow, please edit the title to include Open-source !
numpad0 5 hours ago [-]
These open models are just commercial binary distributions made available at zero cost with intention to cripple opportunities for Western LLM providers to capitalize on investments.
These are more like really gorgeous corporate swags than FOSS.
badsectoracula 3 hours ago [-]
> intention to cripple opportunities for Western LLM providers to capitalize on investments.
Western LLM providers release open weight models too (e.g. Mistral).
Blahah 6 hours ago [-]
Why? Open source isn't in the original title
SV_BubbleTime 5 hours ago [-]
Also “open source” I feel covers for “open weights” which is not the same thing.
This throughput assumes 100% utilizations. A bunch of things raise the cost at scale:
- There are no on-demand GPUs at this scale. You have to rent them for multi-year contracts. So you have to lock in some number of GPUs for your maximum throughput (or some sufficiently high percentile), not your average throughput. Your peak throughput at west coast business hours is probably 2-3x higher than the throughput at tail hours (east coast morning, west coast evenings)
- GPUs are often regionally locked due to data processing issues + latency issues. Thus, it's difficult to utilize these GPUs overnight because Asia doesn't want their data sent to the US and the US doesn't want their data sent to Asia.
These two factors mean that GPU utilization comes in at 10-20%. Now, if you're a massive company that spends a lot of money on training new models, you could conceivably slot in RL inference or model training to happen in these off-peak hours, maximizing utilization.
But for those companies purely specializing in inference, I would _not_ assume that these 90% margins are real. I would guess that even when it seems "10x cheaper", you're only seeing margins of 50%.
I assumed it was to slightly correct this problem and on the surface it seems like it'd be useful for big data places where process-eventually is enough, e.g. it could be a relatively big market. Is it?
You can also use duty cycle metrics to scale down your gpu workloads to get rid of some of the slack.
> These two factors mean that GPU utilization comes in at 10-20%.
Why don't these two factors cancel out? Why wouldn't a company building a private GPU cluster for their own use, also sit a workload scheduler (e.g. Slurm) in front of it, enable credit accounting + usage-based-billing on it, and then let validated customer partners of theirs push batch jobs to their cluster — where each such job will receive huge spot resource allocations in what would otherwise be the cluster's low-duty point, to run to completion as quickly as possible?
Just a few such companies (and universities) deciding to rent their excess inference capacity out to local SMEs, would mean that there would then be "on-demand GPUs at this scale." (You'd have to go through a few meetings to get access to it, but no more than is required to e.g. get a mortgage on a house. Certainly nothing as bad as getting VC investment.)
This has always been precisely how the commercial market for HPC compute works: the validated customers of an HPC cluster sending off their flights of independent "wide but short" jobs, that get resource-packed + fair-scheduled between other clients' jobs into a 2D (nodes, time) matrix, with everything getting executed overnight, just a few wide jobs at a time.
So why don't we see a similar commercial "GPU HPC" market?
I can only assume that the companies building such clusters are either:
- investor-funded, and therefore not concerned with dedicating effort to invent ways to minimize the TCO of their GPUs, when they could instead put all their engineering+operational labor into grabbing market share
- bigcorps so big that they have contracts with one big overriding "customer" that can suck up 100% of their spare GPU-hours: their state's military / intelligence apparatus
...or, if not, then it must turn out that these clusters are being 100% utilized by their owners themselves — however unlikely that may seem.
Because if none of these statements are true, then there's just a proverbial $20 bill sitting on the ground here. (And the best kind of $20 bill, too, from a company's perspective: rent extraction.)
A company standing up this infrastructure is presumably not in the business of selling time-shares of infrastructure, they're busy doing AI B2B pet food marketing or whatever. In order to make that sale, someone has to connect their underutilized assets with interested customers, which is outside of their core competency. Who's going to do that?
There's obviously an opportunity here for another company to be a market maker, but that's hard, and is its own speciality.
A node of 8 H100s will run you $31.40/hr on AWS, so for all 96 you're looking at $376.80/hr. With 188 million input tokens/hr and 80 million output tokens/hr, that comes out to around $2/million input tokens, and $4.70/million output tokens.
This is actually a lot more than Deepseek r1's rates of $0.10-$0.60/million input and $2/million output, but I'm sure major providers are not paying AWS p5 on-demand pricing.
Edit: those figures were per node, so the actual input and output prices would be divided by 12.$0.17/million input tokens, and $0.39/million output
An H100 costs about $32k, amortized over 3-5 years gives $1.21 to $0.7 per hour, so adding in electricity costs and cpu/ram etc... runpod.io is running much closer to the actual cost compared to AWS.
And what stinks is that you can't even build a Dell/HPE server like this online. You have to 'request a quote' for an 'AI Server'
Going through SuperMicro, you're looking at about $60k for the server, plus 8 GPU's at $25,000 each, so you're close to $300,000 for an 8 GPU node.
Now, that doesn't include networking, storage, racks, electricity, cooling, someone to set that all up for you, $1,000 DAC cables, NVIDIA middleware, downtime as the H100's are the flakiest pieces of junk ever and will need to be replaced every so often...
Setting up a 96 H100 cluster (12 of those puppies) in this case is probably going to cost you $4-5 million. But it should cost less than AWS after a year and a half.
Reversing out these numbers tells us that they're paying about $2/H100/Hour (or $16/hour for a 8xH100 node).
Disclaimer (one of my sites) https://www.serversearcher.com/servers/gpu - says that a one month commit on a 8XH100 node goes for $12.91/hour. The "I'm buying the servers and putting them in COLO rate" usually works out at around $10/Hour, so there's scope here to reduce the cost by ~30% just by doing better/more committed purchasing.
Inference is more profitable than I thought.
Is that just the cost of electricity, or does it include the cost of the GPUs spread out over their predicted lifetime?
Maybe the cost of renting?
That's silly, but the idea that "local" is not the opposite of remote is even sillier.
Lots of people were advocating for running their k8s on bare metal servers to maximize the performance of their containers
Now wherever that's applied to your conversation... I've no clue, too little context ( ｡ ŏ ﹏ ŏ )
Bare metal in the context of running software is a technical term with a clear meaning that hasn't become contested like "AI" or "Crypto" - and that meaning is that the software is running directly on the hardware.
As k8s isn't virtualization, processes spawned by its orchestrator are still running on bare metal. It's the whole reason why containers are more efficient compared to virtual machines
Of course, a process running inside Kubernetes Pod, on a baremetal node will show up in `top` if I run it on the node directly. In such terms, it is running directly on hardware.
But when I deploy this Pod, I'm not interacting with the OS in any way. I'm interacting with Kubernetes apiserver, telling it what to run, not really caring about the operating system underneath. In such terms, the application is running "in k8s".
https://en.m.wikipedia.org/wiki/Bare-metal_server
It also does not always "clearly" have this new meaning. Somebody who is used to running programs directly (with no intermediate OS) on hardware might not understand what you're saying, or might ask you to clarify, and you probably shouldn't feel put upon by a totally understandable misinterpretation.
edit: Especially when you keep repeating "directly on hardware" when you mean "not on a VM." VMs also run on hardware. You're saying that you're only running on one OS instead an OS in your OS.
I a Java app running on Linux bare metal?
I consult in this space and clients still don't fully understand how complex it can get to just "run your own LLM".
Just in case you have $3-4M lying around somewhere for some high quality inference. :)
SGLang quotes a 2.5-3.4x speedup as compared to the H100s. They also note that more optimizations are coming, but they haven't yet published a part 2 on the blog post.
These are more like really gorgeous corporate swags than FOSS.
Western LLM providers release open weight models too (e.g. Mistral).